Article Text

Download PDFPDF

Comparing the effectiveness of a hybrid simulation/lecture session versus simulation alone in teaching crew resource management (CRM) skills: a randomised controlled trial
  1. Roberto L Mempin,
  2. Wendy M Simon,
  3. Jason D Napolitano,
  4. Rachel P Brook,
  5. Owen L Hall,
  6. Sitaram Vangala,
  7. Edward S Lee
  1. Department of Internal Medicine, Ronald Reagan UCLA Medical Center, Los Angeles, California, USA
  1. Correspondence to Dr Roberto L Mempin, Department of Internal Medicine, Ronald Reagan UCLA Medical Center, Los Angeles, CA 90095, USA; rmempin{at}


Introduction Various methods have been used to teach crew resource management (CRM) skills, including high-fidelity patient simulation. It is unclear whether a didactic lecture added on to a simulation-based curriculum can augment a learner’s education.

Methods Using an already existing simulation-based curriculum for interdisciplinary teams composed of both residents and nurses, teams were randomised to an intervention or control arm. The intervention arm had a 10 min didactic lecture after the first of three simulation scenarios, while the control arm did all three simulation scenarios without any didactic component. The CRM skills of teams were then scored, and improvement was compared between the two arms using general estimating equations.

Results The differences in mean teamwork scores between the intervention and control arms in scenarios 2 and 3 were not statistically significant. Mean scores in the intervention arm were lower than in the control arm (−0.57, p=0.78 for scenario 2; −3.12, p=0.13 for scenario 3), and the increase in scores from scenario 2 to 3 was lower in the intervention arm than in the control arm (difference in differences: −2.55, p=0.73).

Conclusions Adding a didactic lecture to a simulation-based curriculum geared at teaching CRM skills to interdisciplinary teams did not lead to significantly improved teamwork.

  • crew resource management (CRM) skills
  • medical education
  • simulation

Statistics from


Evidence suggests that teamwork interactions1 2 and group cognitive processes3 play a substantial role in contributing to patient safety, and there is a definite need for further research in crew resource management (CRM) training in the healthcare setting.4 5 The Agency for Healthcare Research and Quality has responded to this need by developing the TeamSTEPPS Patient Safety Program, which was originally adapted from training materials from US Army Aviation and the US Department of Defense.6 This 7-hour multidisciplinary team programme has proven effective at teaching teamwork principles and enhancing provider communication,7 but questions remain about more effective and cost-efficient modalities at teaching this teamwork curriculum.

Various methods have been used in medical education to teach teamwork skills, including classroom lectures, computer sessions, small group problem-based learning and high-fidelity patient simulation. Simulation training is an especially attractive modality due to its standardisation and repetition of content, interactive learning without patient risk and the ability to design goal-oriented clinical experiences.8 However, the utility of simulations in learners without proper background knowledge is debatable. For example, a review of the effectiveness of high-fidelity simulation in undergraduate nursing students found improvements in knowledge acquisition but a lack of clear evidence showing improvements in clinical reasoning.9 This finding in novice learners is not entirely surprising. Miller’s pyramid of clinical competence suggests that novice learners are still working on knowledge acquisition. It cannot be assumed that young learners will develop the appropriate behavioural changes without explicit instruction. Given that most clinicians have not undergone formal teamwork and communication training, it is unclear if high-fidelity simulation alone is the ideal way to teach these skills.

In this study, our group evaluated the effect of a lecture on a simulation-based curriculum to improve the teamwork and communication skills of a multidisciplinary code team. Code blue adult cardiac arrest teams were randomised to a lecture arm or control arm, and CRM skills were measured and compared.

Our hypothesis was that a lecture would augment the simulation-based training and teams that received the intervention would show improved teamwork and communication skills after receiving the intervention as compared with the control arm.


The study was conducted from August 2015 to July 2016 over 22 study days (approximately 4 hours each) at a university simulation centre. There is an existing simulation-based curriculum at our institution for the code blue adult cardiac arrest teams, which includes internal medicine residents, anaesthesiology residents and nurses who have been in place for 5 years. The goal of this curriculum is to improve the teamwork skills of the residents and nurses, specifically their skills in effective communication, conflict resolution, information exchange and maintaining a shared mental model. Multidisciplinary teams were randomised into two arms: one arm acted as a control and participated in three simulation scenarios with debriefing sessions, while the intervention arm received an approximately 10 min lecture given by a hospitalist working as a clinical educator facilitating the session after the first simulation scenario. Following the lecture, the intervention arm participated in the second and third simulation scenarios. Both arms covered the same clinical scenarios.


Two members of the staff at the simulation centre who have no supervisory or evaluation role over any of the residents or nurses conducted the recruitment and consent before the simulation scenarios began.

Simulation scenarios

The METI (Medical Education Technologies, Sarasota, FL) Human Patient Simulator is a computer-controlled manikin that can mimic various physical findings and conditions. It features pulses, breath sounds, heart tones and gas exchange. The manikin interfaces with patient monitors and equipment and allows modelling of an extensive number of diseases complemented by realistic haemodynamic and respiratory responses to drugs and interventions.

After a brief orientation to the simulation environment, the manikin and the available supplies, the multidisciplinary team took part in three scenarios during each study day (see figure 1). Teams were composed of three to four internal medicine residents, one to two anaesthesiology residents and three to five nurses. All residents were postgraduate year 2 (PGY-2) and above, with each team having a variety of different training levels. Each scenario began with a simulation staff member providing information to a small portion of the team (one to three members). This portion of the team would have to quickly evaluate, attempt to diagnose and/or treat the patient, and eventually make the decision to call the code team (the rest of the team members). The scenarios were an intracerebral haemorrhage due to a supratherapeutic international normalised ratio, cardiogenic shock due to myocarditis and right ventricular failure due to volume overload in a patient with pulmonary hypertension. This kind of structure provides the opportunity to evaluate the decision of when to call a code blue, communication handoffs and the transition of leadership in response to changing circumstances. Actors who take the role of a nurse ensure the simulation scenario flows smoothly and does not veer away from the intended course of action as there are no built-in safety errors. Each scenario took approximately 20–25 min, and a clinical educator facilitated a debriefing session after each scenario.

Figure 1

Schematic of study design. n, number of code teams.

All clinical educators facilitating the 30 min debriefing session go through a 2-day faculty development course prior to doing any debriefing. The clinical educators receive instruction similar to the Promoting Excellence and Reflective Learning in Simulation format of debriefing and promote the concept of debriefing with good judgement.10 11 While two debriefing sessions can never be identical due to the need to respond to specific needs of the learner group, every case had clear learning objectives that the clinical educator addressed.

Members of the multidisciplinary team participated in three simulation scenarios with three corresponding debriefing sessions per study day. Each study day had only one multidisciplinary team participating, and no team or team members participated more than once.


Multidisciplinary teams randomised to the intervention arm received an approximately 10 min lecture given by the clinical educator facilitating the session (see online supplementary material). This lecture focused on improving teamwork and communication by presenting tools adapted from the TeamSTEPPS curriculum.6 This included the use of SBAR (explaining the Situation, providing the Background, coming up with an Assessment and giving a Recommendation) during communication handoffs, emphasising a culture of collaboration and speaking up (telling people you are Concerned, Uncomfortable, and you believe this is a Safety issue, or CUS), practising closed loop communication when executing tasks and huddling together to ensure everyone has a shared mental model of the problem. Two of the four authors providing the didactic lecture were trained in CRM skills through TeamSTEPPS courses and help run the TeamSTEPPS course at our institution. The other two authors providing the lecture have received instruction on the key components of this didactic lecture.

Supplementary file 1


All simulation scenarios were videotaped and watched by two reviewers. Five of the authors of this study reviewed videos, and each video was watched by two of the five available reviewers. The reviewers were blinded to whether the session was in the intervention or control arm of the study. Teamwork was assessed by the viewer via a scale adapted from the Mayo High Performance Teamwork Scale (MHPTS), a tool that has been shown to have good inter-rater reliability with naïve raters.12 All 16 items of the MHPTS were included in our rating scale, along with five additional questions which focused on the evaluation of tools used to improve teamwork and communication taught specifically in the lecture. Therefore, our rating scale consisted of 21 items, each of which is rated on 3-point scale (see table 1). For each item, the viewer of the video gave a score of 0 (never or rarely), 1 (inconsistently), or 2 (consistently) in respect as to how often the team displayed each characteristic. Clinical knowledge and technical skills were not explicitly evaluated. Scores were summed and then averaged between two viewers. All video reviewers met prior to the study and evaluated a practice video together in order to ensure a standardised understanding of the terms and definitions.

Table 1

Scoring sheet for teamwork evaluation

Statistical analysis

Mean scores and SDs were calculated for each of the three scenarios across all the multidisciplinary teams in each arm. A linear marginal model, fitted using general estimating equations, was used to make group comparisons. Outcomes were scored at scenarios 2 and 3, adjusting for scenario 1 score. The model included group and scenario fixed effects, a group-by-scenario interaction term and clustered scores by team. Comparisons were made using model contrasts. P values were calculated using an asymptotic z-test, and values <0.05 were considered statistically significant. All analyses were performed using R V.3.4.1 (


All internal medicine residents, anaesthesiology residents and nurses who took part in the simulation-based curriculum participated in the study. Twenty-two teams participated in the study.

Distributions of scores in each scenario for each arm are displayed in figure 2 and summarised in table 2. The mean score of the intervention arm in the first scenario (prior to the intervention) was 22.67 (ΣΔ: 6.36) and 20.15 for the control arm (ΣΔ: 6.21). For the postintervention scenarios, there was no significant difference between the mean scores in the control and intervention arms (−0.57, p=0.78 for scenario 2; −3.12, p=0.13 for scenario 3, see table 3). We additionally compared the change from scenario 2 to scenario 3 in the two arms, but the difference in changes was also not statistically significant (difference in differences: −2.55, p=0.73). Estimated differences suggested lower scores and less improvement from scenario to scenario in the intervention arm, however we cannot rule out that these simply reflect chance variation owing to our small sample size.

Table 3

Marginal model results using general estimating equations (GEE)

Table 2

Mean scores and SDs for simulation scenarios

Figure 2

Box plot showing quartile distribution of scores over the three scenarios compared between the control and intervention groups. Score distributions are further analysed by subdividing into validated scores and non-validated scores.


This is one of the few randomised controlled trials (RCTs) to investigate CRM training in the healthcare setting. Our study demonstrated that adding a didactic lecture to a simulation curriculum did not lead to improvement in CRM skills. In fact, there was a non-significant trend towards improved scores in the control arm compared with the intervention arm, and the control arm appeared to have greater improvement of scores over the three scenarios compared with the intervention arm.

Few healthcare providers receive formal training in teamwork. Curriculum explicitly teaching CRM skills is rarely seen in medical schools and numerous questions remain regarding implementation of a teamwork-based curriculum in graduate medical education and beyond. McCulloch et al showed in a systematic review that there is relatively weak evidence supporting the effect of teamwork and communication training in improving the safety and reliability of clinical care.4 Moreover, innate limitations of teamwork training research, such as the lack of a standardised measurement tool, and multiple extraneous factors which obfuscate the causal relationship between intervention and result further hinder research progress in this field. Despite the difficulties in the research of optimisation of teamwork training, there is no question that CRM skill training remains an important issue. The focus of patient safety improvement has long been in the fields of technology, but the fact that teamwork skills play a huge role in patient safety as well has been recently brought to the forefront.1 2

Simulation-based training is currently used as one of the primary tools for teaching teamwork skills at our institution. Simulation sessions taking place in a controlled simulation facility provide a learner-focused, non-threatening educational environment that is unencumbered by patient service commitments.8 The application of simulation ranges from routine skills and critical event training to competency assessment.8 This modality clearly also has a role in teaching teamwork skills, as Shapiro et al showed a trend towards improvement in the quality of team behaviour when simulation-based training is added to an existing didactic lecture-based curriculum.13 On the other hand, Frengley et al showed no significant difference in teamwork behaviour between simulation-based trained teams and case-based trained teams.14 The results from the literature are decidedly mixed, and more intensive, strict research is needed in this arena. We chose a lecture as our intervention because it can be easily added into our current simulation-based curriculum and questions remain regarding the possibility of a lecture augmenting simulation-based training.

Hobgood et al conducted an RCT randomising fourth year medical students and nursing students into four training cohorts: lecture (control), lecture with audience response, role-play and simulation.7 There was no difference between the four cohorts in teamwork skills when assessing student performance in a standardised patient exercise.7 Clay-Williams et al conducted an RCT as well, randomising doctors, nurses and midwives into four training cohorts: no training (control), classroom-based course, simulation-based training and combination classroom followed by simulation training.15 There again was no difference between the four groups in respect to teamwork behaviour during a patient simulation. Moreover, Lighthall et al randomised multidisciplinary teams to a simulation group and a simulation with lecture group and found no difference in teamwork behaviour or technical skills in a simulation scenario about septic shock.16 The lack of a significant difference between our intervention and cohort groups agrees with the results of these previous studies. However, our study extends on this existing literature by using a validated scoring system and multiple simulation scenarios during each study day to better evaluate the effect of a lecture on a simulation-based curriculum.

There are many possible reasons behind the lack of a significant difference between our intervention and control arms. The lecture material, although taken from the TeamSTEPPS curriculum, may not provide tools that result in a measurable difference in teamwork behaviour. The timing of the delivery of the lecture (between the first two simulation scenarios) may not allow enough time for the information to modify the learner’s mental framework, or the length of the lecture may not have been adequate. Also, the debriefing sessions after each scenario in both the control and intervention arms can dilute the effect of the lecture intervention, as the issues the lecture specifically addresses likely were topics of discussion in the debriefing sessions. Moreover, the fact that a lecture may add nothing to the improvement of communication and teamwork skills is a very real possibility. In fact, our study showed a trend towards more improvement in the control arm, a result seen in the Lighthall study as well.14 There is no clear explanation as to why adding a didactic lecture to a simulation curriculum would lead to decreased scores. Possible hypotheses include that learners may find the lecture distracting and/or unhelpful, and teamwork skills can only be learnt outside of clinical practice via simulation. Additionally, given our limited team sample, we may have been underpowered to detect plausible intervention effects.

The limitations of our study include a small sample size, a single institution and the lack of a gold standard measuring tool for teamwork behaviour. The study also does not make a baseline assessment of CRM skills of each of the code blue teams, as it is assumed that most if not all study participants lack formal TeamSTEPPS training. The strengths of our study include the randomisation and lack of dropout, simulation scenarios taking place before and after the intervention and the multiple simulation scenarios during each study day. The different experience levels of the residents (PGY-2 or PGY-3) and nurses in each team can be seen as a confounding error, but the first scenario prior to the lecture didactic acts as a baseline that controls for differences in training. Furthermore, the timing of the study day (earlier in the academic year with less experienced residents vs later in the academic year with more experienced residents) is controlled for by the random distribution of the control and intervention sessions throughout the year.


Teaching teamwork skills remains challenging but critically important. In this RCT, adding a didactic lecture to a simulation curriculum did not lead to improvement in CRM skills, and there was actually a non-significant trend towards improved CRM skills in the control arm compared with the intervention arm. The lack of significant difference between the intervention and control arms can possibly be explained by the impact of the actual simulation and debriefing sessions in both arms, diluting the effect of the lecture intervention. Further research is needed to define the best teaching modalities for improving teamwork skills among clinical teams.



  • Contributors RLM and ESL conceived the idea, designed the study and organised the simulation sessions. RLM, WMS, JDN, RPB and ESL reviewed the simulations and scored the CRM skills of each team. OLH and SV devised the data analysis and performed the computations. RLM and ESL wrote the manuscript with input from all the authors.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Ethics approval UCLA Institutional Review Board.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.