- Owain Michael Leng1,
- Charlotte Rothwell2,
- Annamarie Buckton3,
- Catherine Elmer3,
- Jan Illing2,
- Jane Metcalf3
- 1 Department of Endocrinology, Royal Victoria Infirmary, Newcastle upon Tyne, UK
- 2 School of Medical Education, Newcastle University, Newcastle upon Tyne, UK
- 3 Department of Medicine, University Hospital of North Tees, Stockton-on-Tees, UK
- Correspondence to Dr Owain Michael Leng, Royal Victoria Infirmary, Newcastle upon Tyne NE1 4LP, UK;
Background The patient safety agenda has propelled the rise of simulation education, but relatively few evaluations of simulation-based educational interventions have focused on patient outcomes.
Objective To evaluate the impact of an in situ, high-fidelity simulation teaching intervention on the management of community-acquired pneumonia in the ambulatory care unit of a district general hospital.
Methods This study used a mixed-methods approach to evaluate the impact of a programme of 10 in situ high-fidelity simulation education sessions delivered to a total of 10 junior doctors, nine nurses and seven healthcare assistants. Participants were tasked with managing a manikin simulating a patient with pneumonia in real time in a working clinical area. Subsequent structured debrief emphasised key themes from the national guidelines on pneumonia management. The intervention was evaluated through an immediate feedback form, follow-up semistructured interviews by independent qualitative researchers that underwent content analysis and triangulation with audit data on compliance with national pneumonia guidelines before and after the simulation intervention.
Results The in situ simulation intervention was valued by participants both in immediate written feedback and in follow-up semistructured interviews. In these interviews, 17 of 18 participants were able to identify a self-reported change in practice following the simulation intervention. Furthermore, most participants reported observing a change in the clinical practice of their colleagues following the training. Collected audit data did not show a statistically significant change in compliance with the guidelines for the management of pneumonia.
Conclusion This study found evidence of a change in both self-reported and observed clinical practice following a simulation intervention, supporting expert opinion that simulation education can impact clinician behaviours and patient outcomes in complex clinical scenarios. Furthermore, this feasibility study provides a transferrable method to evaluate the real-world impact of simulation education that merits further investigation through an appropriately powered study.
- high fidelity
- medical education
Statistics from Altmetric.com
Improving patient safety is the major impetus behind the widespread uptake of simulation-based education (SBE) throughout undergraduate and postgraduate clinical curricula both nationally and internationally.1, 2 However, there is a relative paucity of evaluation of the impact of SBE on clinician behaviour and patient outcomes except for procedural competences,3 despite the considerable financial and staff investment required to establish and maintain centres capable of delivering high-fidelity SBE. Although SBE has a strong theoretical underpinning, coherent with key tenets of constructivist, experiential and adult learning theories,4 a need for more high-quality educational research to justify such investment was recommended in a joint National Simulation Development Project by the Higher Education Academy, the Association for Simulated Practice in Healthcare and National Health Service Health Education England.5
This project uses mixed methods to evaluate the impact of an SBE intervention on clinician behaviours and clinical outcomes related to the management of community-acquired pneumonia (CAP) in a district general hospital. As such, this study’s methods provide an approach to the evaluations of SBE interventions that could be transferrable to other clinical settings or education projects.
In SBE, the term ‘high-fidelity’ is used to describe an interactive, immersive scenario, requiring the same skillset as a real-life occurrence of that scenario.6 7 This can be achieved using advanced computerised manikins to simulate an unwell patient, whose reported symptoms and physiological variables dynamically react to the learners’ interventions, thereby providing the self-generated feedback integral to high-fidelity SBE.8 The simulation scenario is followed by a debrief, using a process of facilitated reflection.9 This debrief is critical to SBE; indeed, simulation without formal debrief seems to be educationally ineffective.10 Through debrief, the issues uncovered in the scenario are explored with open questions, which probe the learners’ rationale and beliefs. The debrief draws on the learners’ prior experiences, contextualising discussions within real clinical experiences, and helps learners to develop plans for improved performance.11
A series of studies have demonstrated the efficacy of SBE in improving clinical outcomes related to procedural competencies, such as bronchoscopy,12 central-line insertion13 and laparoscopic surgery.14 An SBE training scheme for acute management of shoulder dystocia has produced significant improvements in clinical outcomes.15 A meta-analysis of 50 SBE studies measured relevant clinical outcomes related to procedural competencies.3 There is little evidence of the effects of SBE on clinical outcomes in more complex clinical situations, such as the multidisciplinary management of an entire clinical presentation. This may be due to perceived complexity of how to best assess the efficacy of interventions that can address a multitude of issues that span clinical, non-technical and human factor domains. The ability to address the interplay of these domains to improve patient safety has been a purported benefit by proponents of SBE.16 Indeed, the use of SBE to address local patient safety issues was a recommendation of the influential Institute of Medicine ‘To err is human’ report.17
This SBE intervention was intended to address a local patient safety issue, namely the management of CAP, in response to data from audit and clinical governance mechanisms. This study focuses on clinician and team behaviours as a route to improving clinical outcomes, but the value of a systems-based approach to addressing such issues should be acknowledged as an alternative or potentially complementary approach. This study builds on work showing SBE interventions have been associated with increased adherence with local sepsis guidelines in an emergency department setting.18 This study aimed to evaluate the impact of an in situ SBE intervention on the management of CAP in the ambulatory care unit (ACU) of a district general hospital and investigate the feasibility of using routinely collected audit data to provide quantitative evidence for triangulation with qualitative study outcomes.
Participants were under no obligation to partake in the project and signed consent forms before taking part in semistructured interview. The audit data were from an ongoing registered audit with prior approval.
The SBE intervention was developed in response to learning needs identified through two mechanisms. First, through review of local audit data, which identified areas of suboptimal compliance with aspects of the British Thoracic Society (BTS) CAP treatment guidelines.19 Second, to address issues identified through the review of cases of CAP in two of the trust’s clinical governance mechanisms: the Mortality Committee and the Serious Untoward Incident Panel. These had identified cases of delayed escalation and poor communication. An SBE intervention was considered an appropriate modality to address these learning needs, which spanned both clinical knowledge and non-technical skills. A standardised scenario was developed to address these learning needs with input from respiratory specialists (figure 1). The resulting SBE session was delivered multiple times in the ACU and opportunistically targeted staff working in this location. The simulation scenario lasted approximately 20–30 min, with a subsequent debrief lasting ~30 min. The number of SBE sessions was limited by the availability of simulation faculty. In addition, prior to each SBE session, the faculty liaised with the nurse in charge of ACU to determine if there was sufficient capacity to allow the simulation to proceed without compromising patient care.
The hospital’s ACU serves the acute unselected general medical intake, including referrals from primary care, admissions through the emergency department and direct paramedic admissions. This unit was chosen to deliver the SBE intervention for three reasons: first, for the practical consideration that the ACU has predictable periods of spare capacity to host the in situ simulation; second, the unit frequently admits patients with CAP; and third, the clinical staff covering ACU also work in the medical admissions unit, the other main admission route for patients with CAP.
The SBE intervention was delivered by trained faculty with regular SBE involvement (detailed in online supplementary appendix 1). The simulation used a Laerdal SimMan3G manikin programmed to reproduce clinical signs consistent with CAP. The scenario commenced with the manikin arriving at the ACU on a hospital trolley accompanied by a relative (role-played by members of the simulation faculty) and a primary care referral letter. Participants were tasked with managing the manikin as per the normal admission pathway. This included initial routine assessments by healthcare assistants (HCAs) and nursing staff, the measurement of bedside observations, assessment by a junior doctor, the requesting of investigations and the written prescription and administration of treatments. Investigation results and patient responses to interventions were introduced in a realistic time frame, such as a gradual improvement in the oxygen saturation following administration of supplemental oxygen. Further detail on the prebrief given to the participants, and the SBE intervention itself, can be found in online supplementary appendix 2. Although participants of this SBE intervention were not asked to rate the level of realism, this interactive and immersive scenario demanded participants to demonstrate much of the same skillset that the management of a real patient with CAP would require and as such can be described as ‘high-fidelity’ SBE.6 7
Supplementary file 1
Supplementary file 2
Following the scenario, the ‘Debrief Diamond’ tool was used to facilitate reflection on the management of both the simulated patient and more widely on the management of real patients with CAP.20 The debrief explored and addressed the learning needs identified through the clinical governance mechanisms detailed above. A focus was kept on optimal care for such patients and emphasised the treatment principles outlined in both the trust and BTS CAP management guidelines.19 The debrief culminated in discussion on tools and techniques to overcome the barriers to achieving such care.
The effect of the SBE intervention was evaluated through three modalities: audit data evaluating the management of CAP, a qualitative interview several weeks after the SBE intervention and a feedback questionnaire immediately following participation in the SBE intervention.
Data from an ongoing regional rolling audit of CAP management were used to compare achievement of the treatment standards laid out in the CAP guidelines in the month before and at completion of the SBE intervention. The audit standards are derived from the BTS guidelines and as such mirror the focus of the postsimulation debrief. The audit collected data on:
Time to chest X-ray
Time to antibiotics
Appropriateness of antibiotic therapy
Administration of oxygen, if clinically indicated
Documentation of the use of the CURB-65 prognostic score (a six point score, with one point for each of Confusion, Urea >7 mmol/l, Respiratory rate ≥30 breaths/minute, a systolic or diastolic Blood pressure less than 90 or 60 mmHg respectively, and age ≥65 years).
As a feasibility study, this research was not powered to collect a sample of sufficient size to allow for statistical testing for meaningful changes in practice or outcome.
Participants were invited to participate in a follow-up semistructured interview to explore the impact of the training on their practice. These constituted telephone interviews conducted by two independent non-clinical researchers from the field of academic medical education. Interviews lasted approximately 10–20 min and were recorded and transcribed verbatim with the interviewee’s permission. The research philosophy drew from a constructivist approach. This recognises that there is no single reality but that there are multiple views of reality depending on individual perspectives, and the researcher and interviewee construct new meaning from the data. The analyses involved thematic analysis, which seeks to identify understanding and group the data into themes that reoccur in the data. The interviews were analysed using summative content analysis.21 This approach involves identifying certain meaning in the text and counting the frequency of this usage linked back to the research questions. The data were analysed by two researchers who agreed on the codes and the frequencies.
The third means of evaluation was through the use of a feedback questionnaire, which is detailed in online supplementary appendix 3.
Supplementary file 3
Ten teaching sessions were delivered to 10 junior doctors (varying from foundation year 1 trainees to core medical trainees), 9 nurses and 7 HCAs. The participants had all worked together on the ACU prior to the SBE intervention. Nine sessions were delivered in the months of March and April, with one session in May. All 10 of the sessions were delivered in their entirety to all participants with only minimal interruptions (eg, to answer pager).
The pneumonia audit completed at the beginning and the completion of the training programme is presented in table 1. The small sample sizes and assumed lack of power in this feasibility arm of the study means that a statistically significant change in practice is not able to be established.
The participants in the SBE intervention were not all available during the period in which the qualitative interviews were undertaken due to a combination of clinical commitments, shift patterns and annual leave. All available participants were interviewed, with no participants declining this interview. In total, 18 of the 26 participants in the simulation training were interviewed: 10 junior doctors, 4 nurses and 4 HCAs. All bar one (a junior doctor) identified changes in practice that they ascribed to the training. In addition, two-thirds (12/18) reported seeing changes in colleagues’ practice following the simulation training. The responses to the semistructured interviews underwent analysis, and from this, the change in practice themes were developed (table 2).
An aspect common to the identified ‘change in practice’ themes was an improved recognition of the urgency of the management of CAP. One nurse, for example, reported: ‘Before the training everybody was like ah it’s just pneumonia just another chest infection but now after the training you realise it’s NOT JUST pneumonia’. Correspondingly, of the 10 doctors interviewed, half reported they would prescribe oxygen and antibiotics at an earlier stage because of the training.
All participants reported that they had valued the debrief following the simulation scenario. Aspects that were praised included the timing of the feedback, the informal manner of the debrief and that the debrief involved the whole multidisciplinary team engaged in the scenario.
Good - informal from doctor, didn’t feel like being tested which is a good thing. Positive feedback. Liked [the] idea training involved everyone - multi-disciplinary (HCA).
The responses to the post-training feedback questionnaire provided further evidence that participants intended to change their practice following the simulation training and are detailed in appendix 3.
As SBE is increasingly incorporated and mandated in clinical education, with attendant high resource requirement, it is imperative that the impact on clinical outcomes and clinicians’ behaviour is assessed.22 The Kirkpatrick model is a widely used and simple conceptualisation of a hierarchy of methods to evaluate medical education.23 24 Most studies evaluating medical education are limited to gauging the learner’s perception of the teaching or assessing the retention of knowledge or skills.25 Few studies achieve evaluation of the impact of training on clinical behaviour (Kirkpatrick level 3) or clinical outcome (Kirkpatrick level 4).26
The objective of much medical teaching is to improve patient outcomes; therefore, there is a coherence to evaluating an educational intervention by assessing impact upon patient outcomes.27 Practical difficulties that hinder the achievement of high Kirkpatrick-level evaluations are numerous and include the confounding effects of other educational interventions and increasing experience, the impact of other healthcare workers input into patient care and insufficient resources (financial and competencies) to carry out such evaluations.28 The present study used relatively simple mixed methods including self-reported change of behaviour and observation of changes in behaviour of a colleague, as well as routinely collected clinical outcome data to explore the impact of the intervention.28 This method could be adopted and adapted to evaluate similar interventions. Triangulating the results of audit data, follow-up semistructured interviews and post-training questionnaires provided corroboratory evidence to evaluate the impact of the SBE intervention on clinical practice. This is an important standard to which SBE should be held; SBE can only impact on patient safety if it changes clinical practice.
This SBE intervention was based partly on the results of regional audit data. Crude audit data alone may not be sensitive enough to identify the system issues underlying a unit’s ability to meet care standards. The use of information from clinical governance mechanisms, including the Serious Untoward Incident Panel and the Mortality Committee, provided complementary insights into the issues surrounding effective management of CAP. Efforts to better understand the system issues that represent barriers to optimal care may help hone the educational messages of SBE. Such a systems-focused approach may have the added benefit of revealing ancillary measures, such as cognitive aids or automated warnings, which could serve to complement SBE interventions.
The interprofessional nature of the simulation training and its delivery within a clinical area were important aspects of this SBE intervention. The training was workplace based with the potential to more realistically rehearse future real-life care scenarios. There is a tendency to focus SBE interventions onto medical or nursing staff, but in time-critical conditions such as a septic patient with severe CAP, the early actions of HCA staff are essential in the recognition of acutely unwell patients and to expedite their urgent care. The SBE scenario allowed each professional to act within their normal working roles and competencies, potentially allowing an interprofessional learning experience that addressed learning needs relevant for each learner’s role. There is the potential opportunity for further work to evaluate whether such interprofessional SBE leads to a greater appreciation of different roles within the clinical team, whether it can adequately address the varying learning needs of different team members through the same SBE intervention and whether this may lead to more efficient team behaviours.
This study exists in the context of recent national and international quality improvement efforts that have focused on the management of CAP, and of sepsis, reflecting the importance of prompt and appropriate management of these conditions on resultant morbidity and mortality. 29 30 These projects have ranged from educational initiatives to system-level interventions.31 32 Clinical guidelines detail management priorities for sepsis and CAP, but we know that clinicians do not always follow guidelines.33 A clinician’s beliefs, behaviours and priorities will influence the management of their patient, and SBE interventions with facilitated reflection may be well equipped to address these factors.
There were several limitations to the present study. It was a small study of one SBE scenario in one Trust. The training was delivered by a small cohort of simulation faculty, and while a nationally published debrief tool was used, there will be aspects of how the SBE was delivered that will be inherently unique to that team. The follow-up period spanned approximately 3 months; a confounder is the wealth of clinical experience and educational opportunities the participants will have been exposed to in that period, which may have also impacted on their practice in managing CAP or potentially altered their perceptions of the impact of the SBE intervention. However, there were no other new interventions targeting CAP management in the Trust during this period. Furthermore, the use of corroboratory means of evaluation through immediate feedback forms, follow-up qualitative interviews and the audit data ameliorates some of these concerns about potential confounders. The opportunistic use of routinely collected audit data reflects the performance of the whole unit and cannot be attributed solely to those who attended the training. The comparison of audit data from February to May also introduces further possible confounding factors, including the lower incidence of admissions with CAP in May. The choice to look at audit data temporally close to the intervention, rather than comparing with more historical data, aimed to minimise the effect of confounding educational and system interventions regarding the management of CAP and sepsis occurring at local, regional and national levels.
This study has shown how audit data can be used to provide corroboratory evidence for the changes in clinical practice uncovered through qualitative research methods. This merits further investigation through an appropriately powered study to assess the effects of the SBE intervention on the audit outcomes.
The present study of an in situ high-fidelity SBE intervention found evidence of changes in both self-reported and observed clinical practice. This supports other work and expert opinion suggesting that SBE can impact clinical behaviours and patient outcomes in complex clinical scenarios. Furthermore, this study provides a relatively simple and transferrable method capable of evidencing the real-world impact of SBE.
- ↵Geneva: World Health Organization. WHO patient safety curriculum guide for medical schools. 2009 http://whqlibdoc.who.int/publications/2009/9789241598316_eng.pdf (accessed 11 Jul 2016).
- ↵Manchester: General Medical Council. Promoting excellence: standards for medical education and training. 2015 http://www.gmc-uk.org/Promoting_excellence_standards_for_medical_education_and_training_0715.pdf_61939165.pdf (accessed 11 Jul 2016).
- Anderson A ,
- Baxendale B ,
- Scott L , et al
- Joyce B ,
- Weil M ,
- Calhoun E
- Kohn LT ,
- Corrigan JM ,
- Donaldson MS
- Kirkpatrick D
- Kirkpatrick DL ,
- Kirkpatrick JD
Contributors OML designed the simulation education intervention, helped to deliver the education intervention, designed the postintervention feedback form, analysed the feedback form data and drafted and revised the paper. He is guarantor. CR conducted telephone semistructured interviews with participants, undertook thematic analysis of this data and contributed to the drafting and revision of the paper. AB helped deliver the education intervention, collected audit data on pneumonia outcomes and revised the draft paper. CE provided specialist advice to develop the simulation education intervention, oversaw the audit data collection and revised the paper. JI advised on the process of the semistructured interviews and thematic analysis, oversaw the semistructured interviews and revised the paper. JM developed the study’s aims, advised on the development of the method and contributed to the drafting and revision of the paper.
Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests CE has received funding from Pfizer for three smoking cessation talks.
Ethics approval The project was approved by the North Tees and Hartlepool NHS Foundation Trust Research and Development Manager through the local Trust process. It was determined that Caldicott approval was not required for this study.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Further details of regional audit data may be obtainable through contacting CE.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.