Background Providing simulation training directly before an actual clinical procedure—or ‘just-in-time’ (JiT)—is resource intensive, but could improve both provider performance and patient outcomes.
Objectives To assess the effects of JiT simulation training versus no JiT training on provider performance and patient complications following clinical procedures on patients.
Study selection We searched MEDLINE, Cochrane Library, CINAHL, PsycINFO, ERIC, ClinicalTrials.gov, simulation journals indexes and references of included studies during October 2014 for randomised trials, non-randomised trials and before-after studies comparing JiT simulation training versus no JiT training among providers performing clinical procedures. Findings were synthesised qualitatively.
Findings Of 1805 records screened, 8 studies comprising 3540 procedures and 1969 providers were eligible. 5 involved surgical procedures; the other 3 included paediatric endotracheal intubations, central venous catheter dressing changes, or infant lumbar puncture. Methodological quality was high. Of the 8 studies evaluating provider performance, 5 favoured JiT simulation training with 18–48% relative improvement on validated clinical performance scales, 16–20% relative reduction in surgical time and 12% absolute reduction in corrective prompts during central venous catheter dressing changes; 3 studies were equivocal with no improvement in intubation success, lumbar puncture success or urological surgery clinical performance scores. 3 studies evaluated patient complications; 1 favoured JiT simulation training with 45% relative reduction in central line-associated blood stream infections; 2 studies found no differences following intubation or laparoscopic nephrectomy.
Conclusions JiT simulation training improves provider performance, but currently available literature does not demonstrate a reduction in patient complications.
- systematic review
- clinical competence
Statistics from Altmetric.com
Healthcare providers from various backgrounds are often expected to master multiple complex procedures. Historically, learners gained these technical skills as described by the ‘see one, do one, teach one’ mantra. While clinical procedures have become increasingly complex in recent decades, the training has remained unstandardised and even haphazard, resulting in growing concern about deficits in learners’ technical competence and the potential relationship to adverse patient outcomes.1
Systematic reviews of simulation training for procedures suggest that it improves procedural performance on actual patients, increases provider confidence and has the potential to reduce patient complications.2–6 Despite these findings, the most effective and efficient methods of providing simulation training remain unknown. Of particular interest is the issue of ideal timing of both initial and refresher simulation training. Currently, teacher and learner time constraints dictate that most simulation training is scheduled irrespective to the timing of the actual procedures.
‘just-in-time’ (JiT) simulation training—training that is done immediately prior to the actual procedure—has been proposed to improve the simulation training process.7–10 Although the idea of ‘warming up’ or ‘rehearsing’ before a procedure has shown promise in the sports and music literature,11–13 this technique has not been widely adopted in healthcare. While some studies suggest JiT simulation training may be superior to no JiT training for improving provider performance on clinical procedures, the impact on patient outcomes is less clear and the settings in which it might be of greatest value are not known.7 ,9 ,10 ,14–19 Two recent systematic reviews found ‘warm-up’ or ‘rehearsal’ training led to improved technical performance, but one included mostly low-quality observational studies and just two types of complex procedures,14 and the other included only surgical procedures.20 To date, no comprehensive review has rigorously evaluated the impact of JiT simulation training for procedures on outcomes assessed on actual patients, and across a range of procedures, providers and clinical environments.
We drafted a protocol outlining our planned approach to addressing our research question. We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement guidelines for reporting our methods and findings.21 Our protocol, which remained unchanged, is available on request.
Study eligibility criteria
We included studies that met the following prespecified criteria: (1) the design was a randomised controlled trial, non-randomised controlled trial or before-after study; (2) participants were healthcare providers or trainees who perform clinical procedures; (3) the intervention was JiT simulation training, defined as a technique to replace or amplify real patient experiences with guided experiences22 which occurred within 24 h prior to the anticipated clinical procedure; (4) the control group received no JiT training, which could include simulation training as long as there was no explicit or thoughtful effort to time the training to actual performance of the procedure; and (5) the study included one or more of our prespecified outcomes (see online supplementary appendix S1).
We prespecified two categories of outcomes—provider performance and patient complications; all outcomes had to be assessed based on procedures performed on actual patients. We chose to exclude the studies measuring performance on simulators (T1—educational laboratory), as our goal was to assess the effects of JiT training on higher levels of the Kirkpatrick model, T2 (patient care practices) and T3 (improved patient and public health).23 ,24 Provider performance outcomes included procedure success rate, defined as full execution of all intended procedural steps, scores on validated performance scales, duration of procedure, number of attempts to success and need for corrective prompts. Patient complication outcomes included rates of procedure-specific adverse events and rates of procedural delay. We prespecified clinical procedure success rate as the primary outcome.
We searched MEDLINE (1946–2014), the Cochrane Library (all databases, Issue 10 of 12 October 2014), CINAHL (1984–2014), PsycINFO (1958–2014) and ERIC (1966–2014) using strategies developed with the guidance of two professional librarians. The search themes were ‘Simulation’, ‘Just-in-time’ and ‘Clinical competence’. No limits or language restrictions were used for the MEDLINE, Cochrane Library, CINAHL or PsycINFO searches. Terms were modified as needed for specific databases. Each search was last updated in October 2014 (see online supplementary appendix S2 for full strategies). We searched ClinicalTrials.gov (1997–2014) for unpublished studies. We also reviewed the entire index (since volume 1) of the journals Simulation in Healthcare and Clinical Simulation in Nursing. Reference lists of eligible studies were manually reviewed for additional studies not found by other search strategies.
Study selection, data collection and quality assessment
Two reviewers each screened half of the studies identified by the electronic database searches based on titles and abstracts, excluding only clearly irrelevant records. The remaining studies were distributed among the four members of the study team such that each record underwent independent full-text review by two reviewers. After isolating the eligible studies, two reviewers independently extracted data from each study using a standardised data collection form, which was developed by consensus of four members of the research team (see online supplementary appendix S3 for details) and assessed methodological quality using both the modified Newcastle-Ottawa Scale (mNOS) and the Medical Education Research Study Quality Instrument (MERSQI)22 ,25 ,26 (see online supplementary appendix S4). While the NOS was originally developed for assessing non-randomised trials/observational studies, the scale we used is a ‘mNOS’ which has been used in other published systematic reviews.2 ,3 ,5 ,27 Disagreements were resolved by discussion between primary reviewers; a third investigator was consulted when necessary.
Data synthesis and heterogeneity
Owing to the varied nature of the reported results, we summarised each study's findings for our two outcome categories qualitatively. If a study reported more than one measure of provider performance or patient complications, we reviewed the quantitative findings to determine the predominant finding for the outcome category: favours JiT, favours no JiT training or equivocal. When studies reported multiple outcome measures, we used clinical outcomes rather than knowledge measures to determine the predominant finding. If multiple components of a validated performance scale were reported, we used the overall scoring to establish the outcome category. We assigned the label ‘equivocal’ when the reported finding or findings demonstrated no statistically significant difference between study arms, or when the variation among a set of findings was too great to confidently determine a favoured arm. The quantitative findings that informed the qualitative summary judgements for each study are shown in table 3.
Assessment of reporting biases
In order to informally evaluate for publication bias, we ordered the studies by sample size to assess whether the smaller studies had consistently larger effect sizes.
We prespecified three subgroup analyses based on the following variables: (1) provider level of training (resident vs attending vs nurse), hypothesising that earlier learners would benefit more from JiT training; (2) procedure complexity (bedside vs surgical), hypothesising that more complex procedures would be more subject to degradation of skill over time, as well as have increased risk of complications, and thus benefit more from JiT simulation training; and (3) location of procedure (outpatient vs inpatient vs intensive care unit (ICU)), hypothesising that these locations would be proxies for severity of illness and that sicker patients would have more difficult procedures and higher risk of complications.
We evaluated the impact of methodological quality on our overall summary impressions in three ways. First, for each domain of the two tools, we excluded studies that were deemed high risk of bias. Second, we excluded studies that had total scores less than four on the mNOS or less than the median of the Medical Education Research Quality Instrument previously described in a systematic review.22 Third, we subgrouped the studies according to whether they were randomised trials versus non-randomised study designs.
Description of studies
Results of search
As shown in figure 1, our search strategies yielded 1805 potentially relevant publications. After exclusions based on title and abstract screening, 36 studies required full-text review, of which 8 met our inclusion criteria.7 ,9 ,10 ,15–19 See table 1 for study characteristics.
Two relevant studies were excluded from our review based on study design.28 ,29 Willaert et al28 found that patient-specific rehearsal prior to operation may be helpful in preoperative preparation and operative performance, and Desender et al29 found that patient-specific rehearsal prior to endovascular aortic aneurysm repair to be feasible and to have face validity, but neither study design included a formal no JiT training control group.
Methodological quality of included studies
The general methodological quality of the eight included studies was high based on both assessment tools. Just one study scored below 4 of 6 on the mNOS and none scored below 11 of 15 on the MERSQI (table 1).17 ,18 Given the intervention being studied was training, blinding of the participants was not possible. Blinding of the assessor, however, was performed in five of the studies.7 ,9 ,15–17 Two studies evaluated the intervention on a single surgeon, which limited the representativeness of the findings.17 ,18
Overall provider performance
All eight studies reported at least one measure of provider performance, but each used unique measures for the specific procedure being studied. Measures included procedure success rate, components of validated performance scales including the Objective Structured Assessment of Technical Skill (OSATS)15 and the Vassiliou scale (VAS),9 surgical time, need for corrective prompts and number of attempts for success. Overall, five7 ,15 ,17–19 of the eight studies favoured JiT simulation training compared with no JiT training while three9 ,10 ,16 studies were equivocal (table 2).
Details of provider performance
Procedure success rate
Only two of the eight studies reported procedure success rates, both were non-operative bedside procedures.10 ,16 Kessler et al,16 who evaluated JiT versus no JiT training for paediatric and emergency room residents who perform infant lumbar punctures in the emergency room, reported no significant difference in overall success rates (38% vs 35%, p>0.05) but did report a decreased mean number of attempts (1.4±0.6 vs 2.1±1.6, p<0.01). Nishisaki et al10, who evaluated JiT versus no JiT training for paediatric and emergency room residents who perform endotracheal intubations in paediatric ICU patients, reported no significant difference in intubation success rates based on first attempt (50% vs 62.5%, p=0.44) or overall (57.5% vs 75%, p=0.19; table 3).
Two studies used surgical time as a measure of procedural performance.9 ,18 Lee et al9 found no significant difference for the time to complete certain portions of various urological procedures including laparoscopic radical/partial nephrectomy, dismembered pyeloplasty and renal cyst decortication. Mucksavage et al18reported statistically significant improvements in surgical time comparing JiT versus no JiT training for both partial nephrectomies (227 vs 282 min, p=0.004) and radical nephrectomies (243 vs 290 min, p=0.02; table 3).
Validated performance scales
Four studies, all evaluating operative/surgical procedures, used various validated scales to assess procedural performance.7 ,9 ,15 ,17 Two of these studies, Calatayud et al15evaluating JiT training among surgical residents performing laparoscopic cholecystectomy and Chen et al7evaluating JiT training among obstetrics/gynaecology residents performing laparoscopic hysterectomy and adnexal/tubal ligation, used a modified OSATS to evaluate areas such as handling of equipment and tissue, use of assistance, and knowledge of procedure. Both studies found statistically significant differences in final OSATS scores favouring JiT over no JiT training (28.5 vs 19.3, p=0.042 and 22.5 vs 19.5, p≤0.001, respectively, both out of 30 possible points7 ,15). Two studies used sections of the VAS score.7 ,9 Chen et al7used sections related to laparoscopic-specific dexterity, efficiency and tissue handling and found statistically significantly higher mean scores with JiT versus no JiT training (18 vs 15.5 out of 25 points, p≤0.001). Lee et al9 evaluated JiT training among urology residents performing laparoscopic radical and simple pyeloplasty and cyst decortication; the study considered only the suturing and knot-tying portion of the scale and found no statistically significant difference between JiT and no JiT training (3.50 vs 3.54, maximum score not provided, p=0.92). Lastly, Moldovanu and colleagues evaluated JiT training for an attending physician performing laparoscopic cholecystectomy and combined elements of the OSATS and VAS to make a Global Rating Scale ranging from 6 to 30. They found improvement in the overall impression scores which ranged from 1 to 5, favouring JiT over no JiT training based on two evaluator assessments (4.86 vs 4.0, p=0.004 and 4.86 vs 4.37, p=0.059)17 (table 3).
Other provider performance measures
Kessler et al16 also reported significant improvement in analgesia use prior to spinal needle insertion (68% vs 19%, p<0.001) and a non-significant improvement in early (appropriate) stylet removal (69% vs 54%, p=0.06). Scholtz et al19 evaluated JiT training for nurses who performed central line dressing changes and reported the percentage of dressing changes that required corrective prompts by observers was significantly lower following JiT versus no JiT training (9% vs 21%, p<0.001; table 3).
Three of the eight studies reported rates of procedural complications, one of which reported improvement after using JiT simulation training and two of which found no difference (table 2).10 ,18 ,19 Scholtz et al19 reported significant improvement in central line-associated blood stream infections, which were reduced to 2.9/1000 line days after the implementation of JiT training compared with 5.3/1000 line days at baseline (p<0.001). On the other hand, Mucksavage et al18 reported non-significant differences between patients whose attending surgeon performed JiT training prior to laparoscopic nephrectomy compared with patients whose attending surgeon did not perform the training for multiple variables including estimated blood loss, narcotic use postoperatively, length of hospital stay, change in creatinine levels and renal ischaemic time. Nishisaki et al10 reported no significant difference in tracheal intubation associated events, 32.5% vs 20.8% (JiT vs no JiT, respectively, p=0.4).
Subgrouping the studies by provider level found one study evaluated nurses (favoured JiT),19 five studies evaluated residents (two favoured JiT, three equivocal)7 ,9 ,10 ,15 ,16 and two studies evaluated attending (both favoured JiT).17 ,18 Subgrouping by procedure complexity found three studies were done at the bedside (one favoured JiT, two equivocal)10 ,16 ,19 and five studies were done in the operating room (four favoured JiT, one equivocal).7 ,9 ,15 ,17 ,18 Subgrouping by location was not possible as location was often unreported. The number of contributing studies was low and variables differed across studies, making it difficult to isolate the effect of a single variable.
None of our sensitivity analyses based on methodological quality led to changes in our overall qualitative impressions, though the number of contributing studies was low.
After ordering the studies by size, no relationship between sample size and effect size was noted. Just as the varied outcome measures used across different studies precluded quantitative pooling, it did not allow for construction of a funnel plot.
Summary of main results
This is the first review to describe the effect of JiT simulation training using a systematic approach, rigorous inclusion criteria and clinical outcomes. Although quantitative meta-analyses were not possible due to the heterogeneity of procedures and outcome measures, several important qualitative findings emerged. Overall, we found that compared with no JiT training, the use of JiT simulation training was an effective means of improving measures of procedural performance across a range of surgical and non-surgical procedures, performed by a variety of provider types. Unexpectedly, subgroup analyses revealed that for residents performing paediatric intubation and infant lumbar puncture—two low-frequency/high-acuity bedside procedures—improvement in success rates was not seen. Only three studies provided data for patient complications, among which just one found improvement with JiT simulation training. No direct harms of JiT simulation training, such as delayed procedure initiation, were mentioned or described in any of the studies.
Our goal was to evaluate the impact of JiT simulation training for procedures on outcomes assessed on actual patients and thus advance JiT simulation-based knowledge. In order to achieve this goal, we limited our inclusion criteria to only T2 and T3 level outcomes, knowing we would vastly decrease the number of eligible studies. We a priori accepted this anticipated decrease in quantity for an increase in the clinical relevance of the included studies. We believe it is useful to know that there are only eight studies, which evaluated the impact of JiT training on clinical outcomes.
Overall completeness and applicability of evidence
Currently available high-quality studies provide a broad perspective on the impact of JiT simulation training on provider and patient outcomes. However, the ability to draw firm conclusions for any one procedure, training approach, or provider group remains challenging. While the findings were generally consistent across the range of interventions and populations studied, we caution against generalising the findings to procedures and participants beyond those represented in our review. With regard to applicability, JiT training seems to be most appropriate as refresher training, with consideration for short training time and readily available relatively low-cost simulation equipment, we speculate that most interventions would be feasible; however, some of the interventions might be difficult to implement at scale due to the time and/or resources required. Although no delays in procedure initiation were described, we understand in high-acuity high-volume settings delays in care could occur as a result of JiT training in a downstream ‘cascade effect’. While this ‘cascade effect’ has not been described as a consequence of JiT training, utilisers of JiT training must be cognisant of potential consequences the utilisation of time and resources may have on other patients.
Quality of the evidence
The quality and rigour of the included studies were generally strong as indicated by MERSQI and mNOS scores. The directional effect of JiT simulation training on provider performance was relatively consistent; however, the varying nature of the procedures made it impossible to aggregate and summarise the findings in a concise, quantitative manner. The lack of assessor blinding in four of the included studies10 ,16 ,18 ,19 could have made JiT simulation training appear more favourable, as could have the single-surgeon intervention design in two of the studies.17 ,18 It should be noted that secular trends were likely occurring during the Sholtz study, which may have contributed to the observed reductions in central line-associated blood stream infection rates.19
Potential biases in the review process
We performed an extensive search of the literature and conducted all of our detailed screenings in duplicate fashion, with at least one experienced clinician in each pair. While using a single reviewer to perform the title and abstract screening could have resulted in missing one or more eligible studies, we sought to prevent this by only excluding obviously irrelevant studies at this step. The language used to describe the use of JiT simulation training is not standardised, which could have introduced bias by obscuring relevant studies. For example, other terms used to describe interventions similar or identical to JiT simulation training are ‘refresher training’, ‘warm-up’ and ‘rehearsal’. JiT simulation training can also entail a variety of technologies, ranging from computer simulations and virtual reality trainers to physical manikins. Additionally, our reliance on qualitative analysis methods necessarily introduced some subjectivity, as it required us to make decisions about what effects were important. Finally, poor reporting of patient complications limited our ability to draw reliable conclusions.
Agreements and disagreements with other studies or reviews
The fact that all of the eight included studies were published in the past 5 years suggests that JiT simulation training is a relatively novel concept, and the generally favourable findings likely forecast a trend towards increasing use over time. Additional factors likely to increase the use of JiT training include demand for improved efficiency and individuality in learner education, increased efforts to improve patient safety, and recognised need for ongoing refresher education.
The unexpected finding of no benefit for the two high-acuity/low-frequency procedure studies may be related to the fact that these studies focused on learners at the stage of establishing initial competency, while other successful studies were evaluating the impact of JiT training as refresher training or warm-up for procedures on which the providers had already established competency. In a recent article by Sawyer et al,30 an evidence-based pedagogical framework for procedural skill training is presented and discussed where learning is broken down into first a cognitive phase which includes ‘learn’ and ‘see’, followed by a psychomotor phase including ‘practice’, ‘prove’, ‘do’ and finally a maintenance phase. Our systematic review suggests that the current JiT training may be beneficial in the maintenance phase as a refresher but not adequate for the cognitive and psychomotor phases of initial procedural learning.30 The included studies fit into the broader body of evidence supporting the idea that mentally visualising tasks and revisiting learned skills prior to performance can be beneficial.14
We found that JiT simulation training leads to improved procedural performance in a variety of clinical situations, and while a hard link between these improvements and patient outcomes remains elusive, we believe that this evidence supports consideration of JiT simulation programmes and initiatives.
While continued and increased use of JiT training is justified based on procedural performance findings, we strongly advocate for further research including evaluation of patient-related outcomes. More research on the timing and duration of the training is needed to establish benchmarks for training programme design.
The authors would like to recognise and thank the following individuals who contributed generously to this research effort: Kyla Donnelly, MPH; Thomas Mead, MLS; Heather Blunt, MLS.
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Contributors MSB, MDT, JMR, MPC and RJL contributed significantly to all aspects of the study including conceptualisation and design, collection of the data, data analysis, drafting of the initial manuscript and reviewing of the final manuscript. MA and AN contributed significantly to the study including conceptualisation and design, data analysis and reviewing of the final manuscript.
Competing interests AN discloses funded research from the Agency for Healthcare Research and Quality (AHRQ). AN and MA are authors on manuscripts included in this systematic review.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement The authors are happy to share the data collected during manuscript review. They published most of the data collected.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.