Background In situ simulation is an emerging technique involving interdisciplinary teams working through simulated scenarios which replicate events encountered in healthcare institutions, particularly those which are either low frequency or associated with high risk to patients. Since it takes place in the clinical environment, it is ideally suited to improving patient safety outcomes.
Objective To identify and appraise all studies assessing contribution of in situ simulation to patient safety, identify gaps in knowledge and areas for future research, as well as suggesting strategies for maximising its impact on patient safety within an institution.
Study selection Three electronic databases (MEDLINE, PubMed and EMBASE) as well as the Cochrane Library were searched for articles relating to patient safety outcomes in in situ simulation. In addition a subject expert was approached to suggest any additional articles not identified by electronic searches. A total of 1795 abstracts were identified and screened, 35 full articles assessed for eligibility for inclusion and a total of 18 full articles included in the review after unsuitable articles were excluded.
Conclusions In situ simulation can improve real-life patient safety outcomes, with 2 studies demonstrating improved morbidity and mortality outcomes following initiation of in situ simulation. There is good evidence to suggest that its implementation leads to improved clinical skills, teamwork and observed behaviours. Additionally, it is ideally suited to detecting latent safety errors (errors identified within a scenario which, if they had occurred in real life, could have led to a degree of harm occurring to a patient).
- in situ
- patient safety
Statistics from Altmetric.com
In situ simulation is a technique involving interdisciplinary teams working through simulated scenarios in their place of work which replicate events encountered in healthcare institutions, particularly those which are either low frequency or associated with high risk to patients. Its purpose is to provide particular focus on the improvement of both interdisciplinary teamwork and aspects of patient safety.1 The fact it takes place at the very heart of where patient care takes place enables problem solving and improvements to patient safety within the clinical environment and provides an opportunity to ‘identify hazards and deficiencies in the clinical systems, the environment and the provider team’.2
Patient safety can be defined as the avoidance of preventable harm to patients during their care in healthcare institutions. It was identified as a key area for development by the WHO in 2004 with the introduction of its patient safety programme.3 A UK report indicated as many as 10% of hospital patients are affected by an adverse patient safety event during the time in hospital. Commonly, these are the result of errors relating to communication, human factors, leadership, the clinical environment or a combination of these.4 In situ simulation offers the opportunity to address all of these sources of error.
While the rationale and methodology for conducting in situ simulation are described elsewhere,1 ,5 this systematic review is the first to assess its impact on patient safety. Simulation-based education (SBE) has been identified as a key educational strategy to address issues of patient safety in the healthcare environment,6 however, much of the research in this area involves simulation in general with little devoted to the specifics of SBE in an in situ context.
The purpose of this review was twofold; to provide a description of current practice when using in situ simulation as a patient safety tool and to identify evidence justifying its efficacy in this context. Our research question was: ‘What types of patient safety indicators can be assessed using in situ simulation and does it improve patient safety?’ Additionally, we sought to determine: ‘What are the motivating factors for conducting in situ simulation?’, ‘Which specialties and healthcare professionals are involved in in situ simulation?’ and ‘Is in situ simulation combined with other educational strategies when used as a patient safety tool?’
Our objectives for evidence synthesis for this review were not based on any single epistemological standpoint as a variety of study types were included to address the research questions. No submission for ethics approval was made for this review as it did not involve any patients or participants.
The following electronic databases were searched for articles published up to and including April 2015 which related to patient safety outcomes in in situ simulation: MEDLINE, PubMed, EMBASE and the Cochrane Library. In addition a subject expert (MP) was approached to suggest additional articles not identified by electronic searches. Databases were searched by combining the terms ‘in situ’, ‘in situ simulation’, ‘mobile training’, ‘point of care training’, ‘team training’ and ‘technology enhanced learning’, with AND ‘safety’, AND ‘error’, AND ‘patient safety event’ AND ‘clinical error’ with no deviation from this on any databases. Searching was limited to English language articles. No further search limits were applied.
Our aim was to identify all description and justification studies9 involving in situ simulation being used as a patient safety tool, thus all study types were included and studies from any healthcare environment involving any type of healthcare professionals were included. In situ simulation was regarded to be any simulation activity occurring in an actual clinical environment.10
Articles were excluded if they related to other forms of simulation and did not involve any in situ simulation (articles involving interventions combining in situ simulation with other forms of medical education were included). Articles involving in situ simulation but which assessed outcomes not relating to measures of patient safety were excluded. Review articles and conference abstracts were excluded, however, reference lists were assessed for additional articles not identified in the electronic searches. Contact was made with authors of articles where clarification regarding its contents was required.
Citations were reviewed independently by OF and JB. Abstracts were screened according to the inclusion and exclusion criteria. Following the screening of abstracts, full papers of studies meeting inclusion and exclusion criteria were analysed systematically and data were extracted using a manuscript screening tool (see online supplementary material 1) recording (1) type of trial, (2) number of participants, their respective roles and clinical specialty, (3) type of interventions (in situ only or combined with other forms of educational intervention), (4) whether the study was to evaluate a new facility, (5) whether in situ scenarios were designed from actual patient safety events and (6) trial outcome measures and results. If the outcome measure was latent safety threats, the number and type of threats, whether a scoring or grading system was used as well the method by which threats were resolved were all recorded. Any disagreement in the inclusion of studies was resolved by consensus.
For the purposes of grading the evidence level and quality of the justification studies, a pro forma was completed (see online supplementary material 2). Articles were not, however, excluded on the basis of their evidence level or quality. The section of the pro forma relating to evidence level was based on the recommendations of McGaghie et al11 where outcomes in simulation can be considered in terms of ‘translational research’, which in this context assigns a grade to literature ranging from T1 to T3. T1-level evidence is where an effect is demonstrated in the simulation centre, T2 is where an effect in downstream patient care behaviours and practices is demonstrated and T3 is where an effect on patient care or public health is demonstrated. The section of this pro forma relating to the quality of trials was based on the Best Evidence Medical Education Collaboration (BEME) recommendations12 and the work of Reed et al.13
An informal process of thematic analysis14 was used to synthesise evidence from the description studies whereby all authors agreed by consensus on the key themes of each study. These themes were then used to populate a summary table included in the results section and the emerging patterns used as the basis for the discussion. In view of the small number of justification trials and heterogeneous nature of their design and findings, we decided meta-analysis would not be applicable.
A total of 1795 abstracts were identified through databases and other sources (figure 1). Abstracts were screened jointly by two assessors and 35 full text articles assessed for eligibility for inclusion. Reasons for exclusion of full text articles included article content referring to a form of SBE other than in situ simulation or where they were found to be review articles or case studies. The 18 full text articles identified after the exclusion of unsuitable articles fell into two distinct categories; description studies identifying latent safety errors (LSEs) as the main study outcome (13 articles summarised in table 1) and justification studies with a main outcome other than LSEs (5 articles summarised in table 2).
What types of patient safety indicators can be assessed with in situ simulation?
The 13 description studies all assessed LSEs as their main outcome (table 1) and the 5 justification studies (table 2) assessed other indicators. These included technical skills and behaviours (3 studies) and morbidity/mortality outcomes (2 studies).
Latent safety threats refer to errors identified within an in situ scenario which, if they had occurred in real life, could have caused harm to a patient. Of the 13 studies where LSEs were assessed, some studies cited broader outcomes including; shaping institutional policy (1 study), patient and/or participant perceptions of the service (3 studies), providing departmental training (1 study), defining optimal staff roles (1 study), response times to emergencies (1 study) and the number of actual patient safety events after starting a new treatment (1 study).
Methods for recording and grading LSEs were varied. Only two studies used formal scoring or grading systems to classify LSEs, using the ‘Reason's Organisational Accident Model’16 and the Joint Commission's ‘Failure Modes and Effects Analysis’.27 The remaining studies defined their own classification systems. Stated techniques used to resolve LSEs included resolving LSEs during debriefing (4 studies), nominating an accountable person or persons to oversee their resolution (3 studies), internal resolution (2 studies), practice change (1 study), formation of a committee (1 study) and, in the remaining 2 studies, a table was provided giving exact details on the way in which individual LSEs were resolved.
Were appropriate trial methodologies used?
The majority of studies (15 studies) used a prospective, observational study design. Of these, one justification study used historical data to compare mortality and morbidity data following introduction of in situ simulation. Another justification study compared technical skills and behaviours prior to, during and following the introduction of in situ simulation.
A total of three justification studies used a randomised control trial design to assess either technical skills and behaviours or mortality/morbidity outcomes.
Which specialties and healthcare professionals are involved in in situ simulation?
Studies involved a wide variety of acute clinical specialties including anaesthetics/critical care, paediatrics, obstetrics, respiratory medicine, neonatology, surgery, oncology and radiology. All of the studies involved multidisciplinary scenarios comprising anaesthetists, physicians, surgeons, nursing staff, technicians, paramedics, therapists, medical students, pharmacists, operating room staff, safety officers and risk managers.
What are the motivating factors for conducting in situ simulation?
A number of studies (10 in total) were performed either in response to patient safety events in the same institution where the in situ simulation was held, or in preparation for the opening of a new facility. Other reasons for performing the studies were improving teamwork and technical skills (3 studies), to improve morbidity/mortality outcomes (2 studies), starting a new high-risk treatment (1 study) and solely to identify LSEs (1 study).
Is in situ simulation combined with other educational strategies when used as a patient safety tool?
Of the 18 studies assessed, 8 employed in situ as the sole intervention, whereas 10 studies assessed the effect of in situ simulation in combination with another form of educational intervention. Other interventions included workshops, didactic lectures, group meeting and training in the simulation laboratory.
In situ simulation was planned (ie, participants were given forewarning that they would take part in scenarios) in nine studies, unplanned in five studies and it was unclear whether the remaining four studies used planned or unplanned scenarios.
Does in situ simulation improve patient safety?
All 18 studies appeared to indicate either direct or indirect improvements in patient safety. Direct improvements to patient safety from in situ simulation were demonstrated through statistically significant improvements in technical skills and behaviours (3 studies), statistically significant morbidity and mortality outcomes (2 studies) and, demonstration that no actual patient safety events occurred after in situ simulation (1 study).
Indirect evidence of improvements in patient safety were demonstrated in 11 studies which measured LSEs. In addition to recording which LSEs occurred, the majority provided a satisfactory explanation as to how they were resolved, thus it would seem logical, that by rectifying latent safety threats, patient safety would have indirectly improved.
Description studies were not formally assessed for quality given their purpose was to describe the feasibility of in situ simulation and detection of LSEs.
Justification studies were assessed for methodological quality and for the strength of conclusions using a pro forma (completed independently by GF and JB) and the agreed strength of conclusions score included on the summary table (table 2). Disagreements were resolved by mutual agreement. One study was deemed to be poor,29 scoring 1/5, suggesting the results are not significant and thus no clear conclusions can be drawn. Three of the studies were scored 3/5,28 ,31 ,32 suggesting their conclusions can probably be based on the results. One study scored 4/5,30 suggesting results are clear and conclusions are very likely to be true.
The justification studies were also assessed to categorise their focus of outcome using the model of ‘translational evidence’ detailed in the methods section, the outcome of which appears in table 2. Three of the studies28–30 scored T2, showing improved patient care behaviours and practices following the simulation training, and two31 ,32 scored T3, demonstrating actual improved patient care outcomes.
This systematic review has discovered a variety of uses for in situ simulation in the context of patient safety improvement. The studies identified show a wide range of clinical specialties and medical professionals engaging in this form of medical education.
Strengths and limitations of the review
Two of the studies demonstrated significant improvements resulting from in situ simulation at the highest T3 level of evidence. Knight et al31 demonstrated improved survival to discharge in the context of paediatric resuscitation, with 60.9% in the intervention group versus 40.3% in the control group (OR 2.3, 95% CI 1.15 to 4.6). Riley et al32 showed a 37% (p≤0.05) improvement in perinatal morbidity and mortality. Additionally, three studies showed significant improvements in technical skills and behaviours, representing T2-level evidence. Demonstrating gains in improving patient safety is difficult, owing to the time it takes for them to become apparent and the necessary time and resources required to bring about change.1 The fact that data exist at the highest evidence level which supports the use of in situ simulation makes a strong case for its wider adoption.
While many specialties are represented in the literature, there are notable exceptions where no evidence was found, including acute medical and surgical specialties. Given the diversity of the acute specialties shown to benefit from in situ simulation, it would seem that others would certainly gain from the addition of in situ simulation programmes. Furthermore, similar benefits may very well also apply to non-acute and community healthcare systems, though no studies in this area were identified by the review, a potential area for study in the future.
Since most of these studies used in situ combined with other teaching approaches as an intervention, it is difficult to state to what extent improvements were solely attributable to in situ simulation. However, Riley et al study contained three groups; an in situ+didactic teaching arm, a didactic teaching only arm and a no intervention arm. Only the arm where in situ simulation was conducted showed improvements in morbidity and mortality, thus in this key study, in situ would appear to have been the most important intervention. Of the remaining studies, eight used in situ simulation as the sole intervention and all showed benefit in terms of patient safety improvement. A further nine used in situ in combination with another educational technique and showed an overall benefit in improving patient safety outcomes. In all of these trials, in situ simulation was the main intervention when considering the amount of teaching time it involved. It therefore seems reasonable to conclude that the observed patient safety benefit of the trials reviewed in this article derive mainly from in situ simulation.
In situ simulation was planned in nine studies, unplanned in five studies and whether it was planned or unplanned was unclear in the remaining four studies. Ideally exercises should be unplanned. Giving participants forewarning that an exercise will take place allows time to prepare, and arguably to perform better than would have been the case in a real life emergency. Unplanned exercises capture performance which is a truer representation of real world performance in the same situation. However, due to difficulties in organising in situ simulation in busy clinical environments, patient care must take priority and planning exercises in advance may be the only way for them to take place. Additionally, when testing new facilities prior to their opening, planned in situ simulation is the only available method.
In the 13 articles which assessed LSEs as the study outcome, each one employed a different method for recording the type of LSEs detected and only 2 studies used a scoring system to grade the severity of LSEs. This lack of a universal method for recording LSEs raises difficulties in auditing practice both within an institution and between different institutions as already occurs with actual patient safety errors and sentinel events. This could be resolved by wider adoption of scoring systems such as ‘Reason's Organisational Accident Model’ and the Joint Commission's ‘Failure Modes and Effects Analysis’ used in two of the identified studies. A similar UK system is the UK National Patient Agency risk matrix. This is currently used to grade the type, severity and likelihood of recurrence of actual patient safety errors, but could easily be adapted for recording LSEs. This could potentially create a standardised method for direct comparison of LSEs and actual patient safety events.
Another limitation of the description studies was a lack of consistency in the way each LSE was resolved. Only one study22 reported that the LSEs detected during in situ simulation prevented these errors occurring in real life, thus it is difficult to conclude whether their method of resolving LSEs (internal resolution) is the optimal method.
One issue not addressed fully in the studies in this review was whether benefits from in situ simulation are sustained. Miller et al29 reported in their study that the observed improvements from in situ simulation were not seen 2 months after its cessation. However, only 25% of participants in the study had completed in situ simulation, thus to infer it does not lead sustained effects, is in our opinion, not a valid conclusion. In contrast, the mortality and morbidity improvements observed by Riley et al appeared to persist beyond the in situ intervention period (6 months). Although this question remains unanswered, most educators view it as a long-term technique requiring regular participation, such that it becomes an intrinsic part of a department's or institution's on-going patient safety strategy.
The justification studies were as a whole of sound methodological quality. The issues identified in methodology within the studies were generally related to the nature of simulation itself, for example teaching a new skill using either simulation or traditional methods, then comparing the two using simulation to assess the new skill, introducing the potential for confirmation bias. While this is not ideal there are some skills that are difficult to assess in a clinical encounter.
Strategies for maximising its impact on patient safety
On the basis of findings of this review, we would advocate the following strategies when organising an in situ simulation programme as part of a patient safety improvement programme:
All patient safety events occurring in an institution or department should be conducted as in situ simulation exercises as soon as is feasible in order to minimise the chance of them recurring.
In situ exercises should take place prior to new facilities opening or prior to beginning new high-risk treatments.
Consider in situ exercises as part of an induction programme when new staff join an institution/department.
In situ simulation should ideally be unplanned (except when performed prior to new treatments or the opening of new facilities).
LSEs should be recorded and ideally graded using an appropriate scoring system to enable audit to take place.
LSEs should be discussed during debriefing and a responsible person nominated to oversee their resolution.
For the uptake and use of in situ simulation to be further extended, it is imperative that well-designed trials are conducted demonstrating further hard outcomes in its favour. Although the studies identifying LSEs have contributed to our understanding of in situ simulation's role in patient safety, future studies must aim to go a number of steps further; potentially by demonstrating that by detecting and rectifying LSEs, the equivalent real-life patient safety events do not occur and, thereby, institutional morbidity and mortality data can be improved. A standardised approach to recording and grading the severity of LSEs which is directly comparable to scoring systems used in recording patient safety events would facilitate this.
No studies were identified providing any cost–benefit analysis associated with the use of in situ simulation as a patient safety tool. As well as demonstrating efficacy, demonstration of the cost-effectiveness (in particular in comparison to other techniques) of an educational strategy would greatly aid its cause.
In situ simulation is already used across a wide range of acute clinical specialties, usually in response to prior patient safety events or before opening new hospital facilities. It is a multidisciplinary educational tool as evidenced by the broad range of healthcare professionals in the studies identified.
In situ simulation is useful for detecting LSEs, though no uniform method for recording or grading their severity was identified. There is good evidence to suggest that its implementation leads to improved clinical skills, teamwork and observed behaviours. Additionally, there is an indication that it can improve real-life patient safety outcomes, with two studies demonstrating improved morbidity/mortality outcomes following initiation of in situ simulation.
Contributors GF conceived and planned the systematic review, as well as jointly assessed the methodological quality and strength of conclusions of studies. Additionally, GF wrote the manuscript in conjunction with the other authors. JB helped conduct the literature search and screening of abstracts, jointly assessed the methodological quality and strength of conclusions of studies and co-wrote the manuscript. OF helped conduct the literature search and screening of the abstracts and co-wrote the manuscript. MP provided assistance as a content expert and co-wrote the manuscript.
Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.