- http://orcid.org/0000-0003-4384-7693Daniel Goldshtein1,
- Cole Krensky2,
- Sachin Doshi2,
- Vsevolod S. Perelman1,3,4
- 1 SimSinai Center, Sinai Health System, Toronto, Ontario, Canada
- 2 Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada
- 3 Department of Family and Community Medicine, Division of Emergency Medicine, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada
- 4 Schwartz-Reisman Emergency Medicine Centre, Mount Sinai Hospital, Sinai Health System, Toronto, Ontario, Canada
- Correspondence to Dr Vsevolod S. Perelman, Faculty of Medicine, University of Toronto, Schwartz-Reisman Emergency Centre, SimSinai Centre, Sinai Health System,Mount Sinai Hospital, Toronto, ON M5G 1X5, Canada;
Background The use of in situ simulation has previously been shown to increase confidence, teamwork and practical skills of trained professionals. However, a direct benefit to patient outcomes has not been sufficiently explored. This review focuses on the effect of in situ simulation training in a hospital setting on morbidity or mortality.
Methods A combined search was conducted in PUBMED, OVID, WEB OF SCIENCE, CINAHL, SCOPUS and EMBASE. 478 studies were screened with nine articles published between 2011 and 2017 meeting the inclusion criteria for analysis.
Results This review selected eight prospective studies and one prospective-retrospective study. Three studies isolated in situ simulation as an experimental variable while the remaining studies implemented in situ programmes as a component of larger quality improvement initiatives. Seven studies demonstrated a significant improvement in morbidity and/or mortality outcomes following integrated in situ simulation training.
Conclusion Existing literature, albeit limited, demonstrates that in situ training improves patient outcomes either in isolation or within a larger quality improvement programme. However, existing evidence contains difficulties such as isolating the impact of in situ training from various potential confounding factors and potential for publication bias.
- team training
- patient safety
- medical education
Statistics from Altmetric.com
Simulation-based education has been an increasingly applied teaching method with a growing body of supporting evidence.1 There are a variety of techniques, tools and methodologies that can be used within simulation-based education.2 One such modality, in situ simulation-based training, is defined as simulated encounters in the exact setting where they are expected to occur, as opposed to dedicated simulation labs or centres. In situ simulations create an encounter within the real working environment. It is particularly relevant to train teams and/or individuals within an institution. More so, employing in situ simulation may mitigate challenges in constructing a simulation centre by negating the need for dedicated physical space other than storage of the equipment. In the context of healthcare, this has been widely incorporated into education programming ranging from early degree programmes to continuing medical education for various providers including physicians, dentists, nurses and physician assistants.3 In situ simulations are often used when training for medical emergencies or ‘codes’, and correspondingly may be referred to as a mock code.4 Another name for in situ simulations is point-of-care simulations, meaning its scope is not limited to medical emergencies and can be used within the context of other healthcare responses.5
In situ simulation-based training may be an important training modality to improve care delivery in high-pressure situations that demand the coordination of many healthcare providers, actions and resources.6 While it has been demonstrated that knowledge and technical skills are vital to providing competent emergency care, non-technical skills referred to as crisis resource management have also been shown to affect overall outcome, and are improved through in situ simulation training.7–10
Studies have described in situ training improving process measures of care delivery. Process measures refer to properties of the care response itself such as the number of errors, time to intervention and healthcare provider comfort, rather than outcomes of the response.11 Conversely, literature reporting the direct impact of in situ simulation on patient outcomes is scarce.12 The definition of ‘patient outcomes’ in literature is broad but the concepts of morbidity and mortality can be used. They are simplistic in their scope yet are a fundamental starting point when assessing the efficacy of a clinical or educational intervention.13 For example, in the context of cardiopulmonary resuscitation, while certain metrics such as time to compressions and time to defibrillation are established surrogate endpoints to code blue response efficacy, it cannot be assumed that the combination of these process measures leads to improved patient outcomes. In one study investigating the outcomes of cardiopulmonary arrest for 290 patients, 95 were successfully resuscitated; however, only 35 were alive at discharge and may have had significant adverse outcomes that were unreported.14 This trend holds true in similar studies reporting that only 15%–35% of in-hospital arrests survive to discharge.15 16 Process measures for emergency responses are more closely associated with individual resuscitation success, but less so with subacute and long-term patient outcomes such as survival-to-discharge and morbidity. Overall, process measures of in situ simulation are not a substitution for assessing patient outcomes directly.
Previous reviews have shown that simulation-based education has positive effects on patient outcomes but have not demonstrated these findings with in situ simulation-based training in particular.17 18 Other reviews acknowledged the need to further investigate in situ simulation as an educational modality to improve patient morbidity and mortality.3 12 19 However, these studies focused on all levels of Kirkpatrick’s model, looking at both integration of knowledge and measures that result in changes in patient outcomes.20 We felt a more robust review of the up-to-date literature is warranted to identify the impact of in situ simulation training on direct patient outcomes.
This systematic review seeks to determine if there is evidence in the literature of in situ simulation training having a correlation with patient outcomes in the domains of morbidity and mortality.
Six independent online databases were used in this review: PUBMED, OVID, WEB OF SCIENCE, CINAHL, SCOPUS and EMBASE. The major keywords in all searches were: in situ, simulation and patient outcomes. The ‘in situ’ keyword was combined with its relevant synonyms for the complete search query: ((in situ OR mock code OR point of care) AND (simulation) AND (patient outcome)). With PUBMED, OVID and CINAHL, relevant MeSH terms were included within the search query with an OR operation between each major keyword. The separation of keywords ‘in situ’ and ‘simulation’ provided an extra catchment net due to the possible different uses of the expression within relevant studies. The time frame ranged from inception until April 2018. Two independent librarians were recruited to help construct the search query. They were acknowledged but did not meet the criteria for authorship.
Abstracts identified within each search were imported into Covidence,21 a systematic review data management software, and duplicates were removed. Articles were screened independently by two reviewers (DG and CK) with conflicts being resolved through discussion, consensus and input from a third reviewer (SP).
The abstract screening process followed the predetermined inclusion criteria: (1) conducted simulations were explicitly in situ; (2) subjects of the studies must have included healthcare providers or trainees; and (3) studies used direct patient outcomes such as morbidity or mortality as a metric of analysis (result in Kirkpatrick level 420 or improvement in 7I framework22).
Studies that were not chosen matched the following exclusion criteria: (1) simulation was not used as an intervention; (2) simulation used was not in situ; (3) dependent variable of the study was not a direct patient outcome (see above) but rather some sort of indirect measure such as latent safety threats or teamwork management; (4) studies published in languages other than English; (5) secondary research studies such as review articles or systematic reviews, although the authors reviewed references for any missed primary studies; (6) studies from non-peer-reviewed articles, magazine and newsletter publications or online publications, abstracts and conference proceedings, and so-called ‘grey literature’. Abstracts and conference proceedings were excluded due to limited extractable information. The population of interest were licensed healthcare professionals and trainees, including nurses, physicians and students. All patient populations were considered. The primary intervention of interest was the implementation of in situ simulation training alone or as a component of a robust training programme. All primary study designs were included. All years were considered.
Article review process
Forty-five articles then underwent full-text screening resulting in 37 articles being removed due to ambiguous simulation protocols (not clarified whether in situ or at simulation lab or centre), non-English full text, unrelated outcomes and exclusive poster presentations. One primary study which was found while reviewing references from an excluded secondary review article was also included. This resulted in a total of nine studies as seen in figure 1.
During the full-text screening, risk of bias was assessed independently by two reviewers (DG and CK) based on the existing Risk of Bias in Non-randomised Studies of Interventions tool for assessing the risk of bias of interventional, non-randomised studies.23 Any conflicts were discussed and resolved. Risk of bias was determined to be low, moderate or serious depending on factors such as confounding variables, selection criteria, intervention classification and completeness in data reporting.
The extraction of the nine articles was conducted independently by the two reviewers (DG and CK) and verified by a third reviewer (SD) using the Covidence software. The extraction process focused on team composition, choice of outcome, intervention length and data analysis methodology.
Due to the potential variable study designs and methodologies (heterogeneity), a quantitative meta-analysis was not performed. Instead, a narrative synthesis analysis that evaluated effectiveness was conducted.24 25 Our theoretical model was based on the hypothesis that in situ simulation does affect patient outcomes through providing unique opportunities for training in the real clinical environment. Primarily we wanted to see if there is literature evidence for a change in direct patient outcomes through this intervention. A preliminary synthesis then followed through an initial description of included studies. Afterwards, the studies were compared and contrasted based on the characteristics, findings and relationships between the variables. Finally, the robustness and methodological quality of the studies and synthesis is assessing with respect to their limitations and biases.
Of the nine selected studies,26–34 seven were conducted in the USA,26–30 32 34 one in the UK33 and one in India.31 All studies used some variation of a prospective cohort approach, with one study incorporating a retrospective analysis as well.31 The publishing dates range from 2011 to 2017. The spectrum of clinical scenarios for which in situ simulation was performed included code emergencies (n=6),26 27 29 31 33 34 detection of in-hospital deterioration (n=1)30 and obstetrical emergencies (n=2).28 32 The patient populations consisted of paediatric (n=4),26 29 33 34 obstetric (n=2)28 32 and adult inpatients (n=3).27 30 31
The selected studies used different methodological approaches to assess the effectiveness of their variable patient safety interventions (table 1). Only three studies isolated in situ simulation as the primary intervention at onset,26 27 34 while an additional study temporally separated a multitude of interventions, including in situ simulation, to assess for individual effects.32 The majority of studies (n=5) incorporated in situ simulation as part of a broad educational intervention.28–31 33 Seven studies were single arm,26 27 29 31–34 and seven studies used a clearly defined preintervention baseline period for statistical comparisons.27–33 Two studies used other hospital inpatient units as the control group.28 30 Four of the selected studies had intervention periods of 1 year or less,27 29 30 34 with the remaining five studies implementing intervention periods lasting between 2 and 6 years.26 28 31–33
The frequencies of in situ simulation varied, ranging from only once (n=2),27 34 to weekly (n=2),30 33 biweekly (n=1),28 monthly (n=2)26 29 and biannually (n=1).31 One study did not specify the number of in situ simulations performed.32 Debriefing following an in situ simulation exercise was performed in eight studies, using either video (n=4)26 27 29 32 or non-video (n=4) modalities.28 30 33 34
All the selected studies included both nurses and physicians in the in situ simulation intervention. Eight studies26–29 31–34 included the entire emergency code team with only one study focusing exclusively on Postgraduate Year 1 residents and nurses.30 For all studies, turnover rates for participating teams were not clearly reported. Only one study specified the team size and individual composition.33 As well, there were varying reporting methodologies for participation rates as it was generally unclear if participants were involved in one or several in situ simulation exercises.
Five of the selected studies used mortality metrics as the primary measured outcome.26 27 29 31 33 The remaining four studies chose specific morbidity metrics including sepsis, septic shock and acute respiratory failure (n=1),30 methicillin-resistant Staphylococcus aureus infection (n=1)34 and perinatal adverse events (n=2).28 32 In seven of the selected studies, there was a significant improvement in the outcomes of interest.26 28–32 34 One study found no statistically significant improvement in mortality.27 Another study did appreciate a significant improvement in hospital-wide mortality, but the primary scope of the study (paediatric intensive care unit) showed insignificant results.33 Table 2 outlines the intervention and outcome characteristics of selected studies, as well as the results as reported.
The research question posited in this systematic review is whether in situ simulation affects patient morbidity and mortality. The choice of selecting direct outcomes was made to limit the review to the most impactful studies. Only nine studies met the criteria to be included in this review. This relatively small number of studies demonstrates a tendency in literature to focus on the acquisition and improvement of skills and process measures as surrogate markers for improved clinical competency rather than patient outcomes. While conducting this review, articles reporting on these surrogate endpoints were abundant. This finding is congruent with other systematic reviews published in the last several years.9 12 17 18 Recently, however, an increasing number of studies have been published that directly demonstrate the positive impact of in situ simulation training on patient morbidity and mortality. Furthermore, the in situ element of simulation training is a much less explored factor when assessing traditional high-fidelity simulation approaches, contributing to a narrow scope of captured studies.
Limitations of selected studies
Causative effect of isolating in situ simulation
We included all studies that had in situ simulation as an intervention either independently or part of a complex training programme. As simulation-based interventions become more common and training programmes more sophisticated, it is becoming increasingly challenging to isolate the impact of the in situ simulation training as an independent intervention. Moreover, in one study with in situ simulation temporally isolated, while a visual decrease in mortality was seen in the provided figure, no statistical analysis was offered with respect to the isolation of the in situ simulation after introduction of other interventions.32 Thus, while in situ simulation was isolated temporally, its statistical significance is unknown.
In practice, in situ simulation will most likely not be implemented independently and will be a component of a complex training programme. What is critical is that training interventions that include an in situ component have been shown to lead to improved patient morbidity and mortality.
Study methodology limitations
All but two of the selected studies used a single-arm prospective approach that chose a specific baseline period for comparison with the intervention results. While perhaps simpler to develop and implement a single-arm study design, the absence of a control group and subsequent population randomisation limits the validity of the findings. Braddock et al investigated in situ simulation as part of a training intervention in a single hospital department, using other hospital wards as the control.30 However, not all hospital wards are equal, thus this may introduce further confounders. Riley et al randomly assigned three hospitals as a control, didactic only, or didactic along with in situ simulation.28 Similarly, not all hospitals are equal which introduced cofounders. Despite these limitations, this is a prudent first step to increasing the validity of the results. However, further efforts should include randomisation of the study population and larger sample sizes to reduce confounding variables such as secular trends in outcomes.35
The dilemma of study methodology can be traced back to the chosen population of interest. Studies that focused on the code team are hindered by issues such as limited sample size, rotating teams and inability to create control groups. Often the number of in situ simulations is reported, but it is unclear if the same people are participating in each one. The ultimate effect of improved patient outcomes is difficult to attribute to any factor, instead correlations are drawn to an abstract general change in safety culture or safety awareness. A greater focus on documentation of individual participation and team change-over may increase transparency of the effectiveness of the in situ simulation intervention.
All selected studies used an interprofessional approach to training which highlights the importance of investing in whole-team training. This is beneficial as in situ training mirrors real-life emergency codes which are attended by a rotating interdisciplinary team. Teams must rely on each member’s technical and non-technical skills to succeed; therefore, training is imperative for all members. Moreover, one study demonstrated that by inviting non-core team members to in situ training, a decrease in general hospital mortality was demonstrated.33 A possible explanation for this was that general ward nurses and medical trainees who participated in the in situ training brought their educational experiences back to other hospital units. Though only demonstrated by a single study, by expanding training interventions department-wide or even hospital-wide beyond the code team, patient outcomes can be further influenced.
Limitations of current review
This systematic review is limited by exclusion of non-English articles and being unable to contact researchers for additional information. Our search criteria were narrow by design to capture only the studies about in situ simulation deemed most impactful with relation to patient outcomes, which may have limited the scope of our search. A larger limitation of this review is possible publication bias since no studies captured in the search reported negative results. This could suggest the search criteria are too narrow, by focusing entirely on simulations with an in situ component, that few studies are investigating morbidity and mortality as an outcome, or that studies with negative or neutral results are not published. As well, the majority of the selected studies are single-arm pre-post intervention studies, thus the risk of bias is relatively high. Long-term trends are another limitation involved in the before-and-after studies we analysed. For example, it has been shown that a hospital without a paediatric medical emergency team has decreased mortality over time, suggesting that multiple interventions or secular trends in outcomes of cardiac arrest can account for this decrease.35
Recommendations for future studies
Current literature tends to investigate in situ simulation independent of patient outcomes, or as a part of a larger educational initiative. However, the relationship between the two is rarely investigated in isolation. To better explore the potential impact of in situ simulation, future studies could implement protocols that effectively isolate its impact on patient outcomes. A prospective study in an isolated unit such as an emergency department can be used to compare patient outcomes before and after the implementation of an in situ training programme. A resource intensive method could be implemented where all potential members of the response team receive in situ training, or instead implement an intent-to-treat approach where specific members (eg, full-time staff and code leaders only) receive training. The emergency department is a strong candidate for this small-scale trial since it often acts as its own code response team (ie, less responders that require in situ training). This smaller scale pilot project could provide data for cost-benefit analysis of an institution-wide initiative for all code responders. To reduce confounding factors, sites can be chosen that have demonstrated relatively consistent code outcomes over a retrospective time period that is equal to or longer than the planned prospective study period. For example, a 3-year prospective in situ study should take place at an institution where code outcomes have remained consistent for the prior 3 years. Additionally, patient outcomes from other institutions without in situ training can be monitored for relative changes. This would reflect and account for confounding factors such as technological and research advancements. Another study could include participants randomised into either the in situ or traditional arms will elucidate the specific impact of in situ training. Overall, a well thought-out prospective study design specifically focused on evaluating the effect of in situ training on patient outcomes is required to truly reveal its impact; however, paucity in current literature persists as the relationship between the two is rarely the sole outcome being investigated.
The present review discovered evidence in the literature that incorporation of in situ simulation training is statistically correlated with improved patient morbidity and mortality. However, supporting evidence remains limited by the number of studies and an array of confounding factors to grasp the true validity of the findings. To determine the true impact of in situ simulation either independently or as part of a larger training programme, future research should make use of more isolating protocols with fewer confounding factors.
Daphne Horn, Chris Walsh, Lisa Satterthwaite.
- Patterson MD ,
- Blike GT ,
- Nadkarni VM , et al
- Kodate N ,
- Ross A ,
- Anderson JE , et al
- Kirkpatrick DL ,
- Kirkpatrick JD
- ↵Cochrane. Covidence systematic review software, Veritas Health Innovation. Melbourne, Australia.
- Popay J ,
- Roberts H ,
- Sowden A , et al
- Theilen U ,
- Fraser L ,
- Jones P , et al
Contributors DG and CK designed the search query, conducted the search, analysed the data, drafted and revised the paper. SD drafted and revised the paper. SP initiated the project and revised the paper. DG and CK contributed equally to this paper.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Ethics approval The authors did not seek ethical approval as the systematic review did not meet the criteria of human subject research.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.