Article Text

PDF

Frameworks and quality measures used for debriefing in team-based simulation: a systematic review
  1. Ruth Endacott1,2,
  2. Thomas Gale3,4,
  3. Anita O’Connor1,
  4. Samantha Dix2
  1. 1 School of Nursing and Midwifery, Plymouth University, Plymouth, UK
  2. 2 School of Nursing and Midwifery, Monash University, Frankston, Victoria, Australia
  3. 3 Collaboration for the Advancement of Medical Education Research and Assessment, Plymouth University Peninsula Schools of Medicine and Dentistry, Plymouth, UK
  4. 4 Department of Anaesthesia, University Hospitals Plymouth NHS Trust, Plymouth, UK
  1. Correspondence to Dr Ruth Endacott, Medicine, Nursing and Health Sciences, Plymouth University, Plymouth PL4 8AA, UK; ruth.endacott{at}monash.edu

Abstract

Objectives The skill of the debriefer is known to be the strongest independent predictor of the quality of simulation encounters yet educators feel underprepared for this role. The aim of this review was to identify frameworks used for debriefing team-based simulations and measures used to assess debriefing quality.

Methods We systematically searched PubMed, CINAHL, MedLine and Embase databases for simulation studies that evaluated a debriefing framework. Two reviewers evaluated study quality and retrieved information regarding study methods, debriefing framework, outcome measures and debriefing quality.

Results A total of 676 papers published between January 2003 and December 2017 were identified using the search protocol. Following screening of abstracts, 37 full-text articles were assessed for eligibility, 26 studies met inclusion criteria for quality appraisal and 18 achieved a sufficiently high-quality score for inclusion in the evidence synthesis. A debriefing framework was used in all studies, mostly tailored to the study. Impact of the debrief was measured using satisfaction surveys (n=11) and/or participant performance (n=18). Three themes emerged from the data synthesis: selection and training of facilitators, debrief model and debrief assessment. There was little commonality across studies in terms of participants, experience of faculty and measures used.

Conclusions A range of debriefing frameworks were used in these studies. Some key aspects of debrief for team-based simulation, such as facilitator training, the inclusion of a reaction phase and the impact of learner characteristics on debrief outcomes, have no or limited evidence and provide opportunities for future research particularly with interprofessional groups.

  • simulation
  • debriefing
  • validity
  • frameworks
  • interprofessional
View Full Text

Statistics from Altmetric.com

Background

In simulation learning, debriefing—‘a discussion between two or more individuals in which aspects of a performance are explored and analysed with the aim of gaining insights that impact the quality of future clinical practice’1is key, and the skill of the debriefer is the strongest independent predictor of overall quality of simulation encounters.2 In a conceptual paper, Haji et al 3 argued for a distinction between simulation-based and simulation-augmented medical education, with the latter integrating the simulation learning with other educational experiences. This approach also places simulation mainstream, rather than as a special event for the privileged few. While simulation-based education is laudable, simulation is an expensive resource especially when used for small group learning. We therefore need to ensure that learning opportunities are optimised when simulation is used.

Effective interprofessional working is important for standards of patient care and is thought to be highly influenced by the attitudes of healthcare professionals.4–6 However, a report from the Centre for the Advancement of Interprofessional Education highlights that many educators feel underprepared in interprofessional, as compared with uniprofessional, settings and recommends that all facilitators receive comprehensive orientation, preparation and ongoing support for Inter Professional Education (IPE).7 Interprofessional team-based simulation allows learning opportunities within the correct educational and professional context8 and has been shown to improve communication skills and understanding of professional roles.7 However, debriefing interprofessional groups brings its own unique challenges due to learner differences in background, experience and professional identity9 requiring faculty to be trained appropriately to debrief interprofessional issues in an effective manner.8

Dreifuerst10 used concept analysis methods to identify defining attributes of debriefing as it relates to simulation to construct model, borderline and contrary cases and to distinguish between unstructured, structured for critique and structured for reflection approaches to debrief. This is a useful addition to our understanding of debriefing but has yet to be subjected to empirical testing. Previous systematic reviews have focused on the advantages of debrief over no debrief and whether the use of video improves the debrief1 11; however, there is a lack of research exploring the evidence base underpinning decisions about debriefing. The main aims of this study were to identify: (1) frameworks used for debriefing interprofessional and uniprofessional team-based simulations, (2) metrics that have been developed to assess the quality of debriefing and (3) evidence gaps for debrief decisions. The term ‘debriefing framework’ is used to refer to the structure used for the debriefing discussion.

Methods

Design

A systematic review was conducted following the procedures set out by the Centre for Reviews and Dissemination,12 whereby specific search terms are used in database searching and papers are selected based on an explicit inclusion and exclusion criteria. We also undertook hand searching of references and sought to identify records through other sources (eg, Google Scholar) in an attempt to include as many relevant papers as possible in the review. We aimed to identify:

  1. Debriefing frameworks used for team-based (uniprofessional or interprofessional) simulation.

  2. Measures to assess the quality of debriefing.

Search strategy

Four electronic databases were searched in December 2017: PubMed, CINAHL, MedLine and Embase. All peer-reviewed articles published in English between January 2003 and December 2017 were eligible for inclusion. Our preliminary searches identified many papers that were not relevant. This 15-year window was decided on for pragmatic reasons and because no relevant papers providing empirical data regarding team-based debriefing were identified prior to this date. As initial searches had identified excessive numbers of papers with either ‘framework’ or ‘method’ in the title or abstract, we refined search terms and ran a further search using the keywords: ‘Simulation’ AND (‘Debrief* OR Feedback’) AND ‘Evaluation’ AND (‘Quality OR Framework OR Method’).

Empirical studies and framework/development studies were included in the review, providing some form of outcome measure was used. Outcome measures assessed quality of the debriefing and/or performance of participants. All included studies used team-based simulation and examined technical and non-technical skills. Studies not published in English focused on individual debriefing and describing only the quality of the simulation (and not including quality or outcome of the debrief) were excluded.

Quality appraisal

Papers were assessed using the Kmet et al 13 quality appraisal tool. The initial appraisal was conducted by two of the authors, with a third author meeting to discuss any differences in the scoring (RE, TG, AO and SD). Any discrepancies in scoring were discussed until consensus was reached.

Results

A total of 676 citations were screened; the Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow chart summarises the review process (figure 1). Abstracts were reviewed for 253 papers; 41 (6.1%) were found to meet the study criteria after review of titles and abstracts by two authors (RE and AO or RE and SD). There were no disagreements on inclusion of papers. The remaining 41 full articles were interrogated and assessed for eligibility; 11 were excluded (including concept analysis, application of a theoretical framework and commentary papers).

Figure 1

PRISMA flow chart. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

A total of 26 papers met the full inclusion criteria and were appraised. Eight papers were excluded from the data synthesis due to a low-quality appraisal score (<0.60); this is common in narrative reviews to ensure synthesis of papers of suitable and comparable quality and that recommendations for future practice are not based on low-quality evidence.13 Tables 1 and 2 show the quality appraisal scores for the 26 papers reviewed.

Table 1

Quality appraisal scores for quantitative studies

Table 2

Quality appraisal scores for qualitative studies

A total of 18 papers were included: 1 qualitative study, 15 quantitative studies and 2 studies containing both qualitative and quantitative components. The quantitative Kmet scores ranged between 65%–100%; the two mixed methods papers14 15 and the qualitative paper16 scored 85%. Summary of the 18 included studies is provided at table 3.

Table 3

Summary of studies included in the narrative synthesis

Demographics

There were 2013 participants across the 18 studies (range 9–450). Twelve studies were conducted in the USA, 2 of which14 15 contained both qualitative and quantitative components, with the remaining 10 comprising quantitative data only. The remaining quantitative studies were conducted in the UK,17 Switzerland,18 Korea19 and the remaining two in Canada.20 21 The only wholly qualitative paper included in the review was conducted in the UK.16

Seven studies were conducted with interprofessional teams and four of these examined differences between the professional groups.16 18 22 23 Geis et al 22 used simulation to model how a new paediatric emergency department would function and to identify latent safety threats; debriefing was structured and included video review. Changes in workload for different professional groups were analysed as the simulated workload of the department changed. LeFlore and Anderson et al 23 compared two approaches to interprofessional team simulation and debriefing; changes in knowledge test scores and satisfaction with the simulation/debrief were reviewed by professional group. In the Freeth et al 16 qualitative study, some excerpts from interviews identified participants by professional group, but there was no comparison between groups. Kolbe et al 18 found that evaluation of their debriefing model—TeamGAINS—did not differ by job role (nurse or doctor).

Debriefing frameworks

All studies included a structured debriefing framework, mostly tailored to the individual study (see table 4). Five authors used a previously validated framework: the Ottawa Global Rating Scale,20 TeamGAINS,18 Debriefing for Meaningful Learning,24 Structured and Supported Debriefing19 and Guided Team Self Correction (GTSC).25 In 11 studies, outcome measures were used to assess debrief quality (faculty behaviours)14 15 17 18 22–24 26–29 and in 12 studies change in performance following the debrief was measured (participant behaviours).16 18 20–25 30–32

Table 4

Debriefing frameworks and measures used in the 18 studies

Performance measures

The majority of studies (12/18) used some measure of performance to judge the success of the debriefing framework, using a before-and-after design or comparing two debriefing frameworks (table 4). A total of 17 measures were used in the 12 studies (table 4).

Synthesis

All papers were read in full by two authors; a combination of inductive and deductive thematic analysis was used to develop codes and categories to relevant extracts and organise these findings under main thematic headings. These are presented at figure 2. Deductive codes were derived from the review aims and the inductive component allowed codes to emerge from the data. A synthesis of these findings was used to identify key themes.

Figure 2

Evidence and evidence gaps for decisions about debrief.

Several key themes were identified through this synthesis of the findings; two authors discussed these themes until a consensus was reached. These themes were: selection and training of debrief facilitators, debrief model and assessment of debrief. The themes are discussed below; summary of the evidence, and evidence gaps, for each theme is presented at figure 2.

Selection and training of debrief facilitators

Most of the studies were conducted with a trained debrief facilitator15–18 22 24 26 29 31 32 with one research team reporting use of ‘PowerPoint plus audio’ with no indication whether the ‘audio’ was prerecorded or provided by a facilitator.14 An randomised controlled trial compared two approaches to debrief: within-team debrief, with a leader from within the team providing the debrief, and instructor-led debrief.20 Team performance, assessed using the Team Emergency Assessment Measure (TEAM),33 improved following debrief in both groups (F 1,38=7.93, p=0.008); there was no significant difference between within-team or instructor debrief (F 1,38=0.43, NS p=0.52). Oikawa et al 32 found that self-debriefing was as effective as faculty debriefing in improving self and team performance assessment across four sequential scenarios.

Different study designs make it impossible to state that one type of facilitator is superior; performance in individual studies improved when the team leader,20 instructor,15 faculty32 or team member32 led the debrief. Similarly, no studies provided evidence that training actually makes any difference.

Debrief model

The format of debriefing reported in the studies varied in three areas: degree of structure, use of video clips and timing of the debrief.

All authors described a debrief framework, with variation in the detail provided. Three authors specify an initial reaction stage (‘how was that for you?’), followed by attention to technical and/or non-technical skills and how they were performed in the simulation scenarios; Lammers et al 15 and Van Heukelom et al 27 refer to this first stage as ‘decompression’, while Kolbe et al 18 describe it as ‘reactions’. No one structure was used across studies; most authors tailored an existing debrief framework.

Training faculty to use GTSC to structure the debrief had a significant impact on overall team performance, over traditional debrief methods (t(11)=1.98, p=<0.05 (one tailed)).25 The group receiving GTSC also developed mental models more similar to those developed by an expert group. In a pretest and post-test study paediatric emergency medicine fellows were trained to use a cardiac arrest debriefing model (REFLECT) with teams of four. The fellows and team members reported significant improvement in use of REFLECT components (63 vs 82%), but blinded expert reviewers reported a non-significant improvement (60 vs 76%).29

Use of Cognitive Disposition to Respond (CDR) to structure the debrief, with technical/knowledge based debrief as the control, resulted in higher satisfaction scores for the technical/knowledge based debrief. This did not reach significance.14 LeFlore and Anderson23 compared a facilitated debrief (group A) with a modified debrief (group B) in which time for questions was allowed. However, the learning interaction was also different with group A using self-directed learning and group B observing experts completing the scenario. Group B had higher satisfaction scores, but there is no indication whether this was due to the expert modelling or the modified debrief.

Video clips were included in the debrief in seven of the studies,15 16 20–23 26 but extent of video use described by the authors was variable. In one study, the researchers compared no debrief (control) with oral debrief (intervention 1) and oral plus video debrief (intervention 2) using a pre–post design with anaesthesia residents.21 There was significant improvement in total Anaesthesia Non-Technical Skills (ANTS) score (F 2,39=6.10, p=<0.005) and scores in each of the four domains for both intervention groups but no significant difference between oral and oral+video groups on total or individual domain scores. Similarly, a pretest and post-test study comparing video-assisted debrief with oral debrief alone with nursing students reported a higher mean score on behaviour for those in the video-assisted debrief group than the control group (6.62 vs 4.23), but this did not reach significance.30

In most studies, debriefing was conducted at the end of the simulation exercise; the one exception was the study conducted by Van Heukelom et al,27 who compared insimulation debrief (identifying learning points and errors as they arise during the simulation) and postsimulation debrief. They report that self-reported confidence and knowledge improved for both groups (Spearman’s R=0.5 with p≤0.001 for all results) with no significant difference between groups. However, the postsimulation debrief group had significantly higher scores for three items on the debriefing satisfaction scale. In seven studies, participants completed a further simulation scenario following the debrief20–25 30; this is reviewed in detail below.

The studies reviewed provide evidence that debriefing frameworks can improve outcomes; however, there is no evidence that including a reaction phase or using video makes any difference to outcomes.

Assessment of the debrief

There were two approaches to assessment of debrief: assessment of debrief quality and change in performance following the debrief.

The quality of the debrief was assessed through satisfaction scores or through analysis of debrief videos. Satisfaction was rated by participants14 23 24 27 28 or faculty,26 or both.17 18 29 Kolbe et al 18 also measured psychological safety and leader inclusiveness before and after the debrief and found both measures significantly improved (t(59)=−2.26, p=0.028 and t(60)=−2.07, p=0.048). In four studies, analysis of debrief videos was conducted using an existing tool: Brett-Fleegler et al 26 used the Debriefing Assessment for Simulation in Healthcare (DASH) with 114 simulation instructors to test validity and reliability, and Lammers et al 15 used a Root Cause Analysis (RCA) framework to examine the quality of RCA processes in a simulated prehospital paediatric emergency. Hull et al 17 used Objective Structured Assessment of Debriefing (OSAD) with expert debriefing evaluators and faculty debriefing, and Zinns et al 29 used the REFLECT postresuscitation debriefing framework.

Significant improvement in performance following debrief was reported in several studies. Change in performance was assessed using: (1) a (different) simulation scenario conducted after the debrief,20–23 (2) participant knowledge, assessed using a pre/post knowledge test,25 (3) participant self-reported confidence and knowledge27 and (4) mental model accuracy.25

The postdebrief simulation performance was assessed using a range of existing measures: the Mayo High Performing Team Scale,22 the TEAM,20 ANTS,21 Behaviour Assessment Tool, based on CRM principles and validated in previous studies by the authors,23 the Health Sciences Reasoning Test,24 Team Dynamics31 and Team Clinical Performance.31 In the Geis et al study,22 the phase 1 (predebriefing) simulation was conducted in the simulation lab, and the phase 2 (postdebriefing) was conducted in the hospital, hence change in behaviour could not be attributed solely to the debrief.

Despite some studies using more than one performance measure, none of the studies reported correlations across performance measures. Where performance data were analysed in the context of demographic data items, these were mainly limited to professional group16 18 22 23 and work experience.

Discussion

There was little commonality across the papers in terms of participants, experience of faculty and measures used; however, all studies used a debriefing framework to provide structure for the debriefs often underpinned by theoretically derived methods to facilitate interaction of participants. Eighteen different debriefing frameworks were described, showing divergence in preferred debriefing techniques and strategies among the studies, but the frameworks commonly started with a ‘reaction’ or ‘decompression’ phase to encourage self/team reflection. The reaction phase assumes that participants will ‘let off steam’ during the first few minutes of a simulation debrief, which provides facilitators with content that should be discussed at some stage in the debrief but also allows participants to express their emotions straight away and provide a more balanced environment for objective reflection later in the debrief.18 None of the studies compared this reaction phase with no reaction phase so the impact is unknown. All debriefing frameworks covered either technical or non-technical aspects, or both and some studies compared participant reactions to either technical/non-technical aspects. Non-technical skills were addressed through the use of expert models such as crisis resource management principles or through techniques such as CDR and Advocacy Inquiry (AI) aimed at identifying mental models of participants, which lead to certain behaviours.14 26 Bond et al 14 found that technical debriefing was better received by participants than cognitive debriefing, although Dreifuerst34 reported that learners prefer debrief with reflection.

The debriefing model described by Kolbe and colleagues18 reflects the recommendations of several earlier authors and comprises six steps: reactions; debrief clinical component; transfer from simulation to reality; reintroduce the expert model; summarise the debriefing; and practice/improve clinical skills as required. This model, as a whole, was shown to have some benefits but our review has shown varying degrees of evidence for each of these steps, as illustrated in figure 2.

Debriefing theory

Different techniques are used to focus the debrief on individuals and team members as well as observers. Debriefing models utilised a range of theoretical techniques to facilitate interaction of the whole group through guided team self-correction, peer assessment, self and team reflection.18 23 25 30–32 Guided team self-correction and circular questioning18 25 are techniques that switch the focus to the whole team and encourage active participation and reflexivity from all members of the group. Smith-Jentsch et al developed the technique of GTSC, where members of the team are responsible for identifying their own team performance problems plus process-orientated goals for improvement.25 In GTSC, an expert model of teamwork is used as an organisational framework at the briefing and then debriefing stages when participants are asked to discuss both positive and negative examples of each component. Debriefing theory developed by Salas and colleagues makes the assumption that the use of an expert model provides a common language for participants to use during team debriefs, which helps to form shared team mental models that match the expert framework.25 35 Reflecting on both positive and negative examples of behaviour has been found to develop stronger mental models and focusing on a few critical performance issues to identify learner ‘process orientated goals’ helps to ensure that learning is not scenario specific. High-level facilitation allows participants to contribute to the majority of discussion in the debrief, which maximises individual reflection and team based reflexivity so that the learners are reaching a deeper level of understanding about the interactions which have taken place, rather than listening to expert opinion by the debriefer. With techniques such as GTSC, the debriefer facilitates from a non-judgemental perspective without expressing their own expert opinion until the latter stages of the debrief, if at all.

In contrast, AI is more instructor led where the debriefer will highlight a performance gap encountered by an individual during the simulation and use direct questioning to uncover underlying mental frames that led to certain actions or behaviours.18 26 The conceptual framework and underlying theory assumes that by exploring the mental frames or thought processes that have led to certain behaviours, the learner is able to rewire these thought processes for similar situations in the future, resulting in different actions or interactions.36

A central tenet across debriefing theories for teams is the development of a shared understanding across participants and facilitator. However, the seven studies we reviewed that were conducted with interprofessional teams did not appear to test mental model consistency across professions.

Learning environment

Creating the right environment has been eloquently described as a ‘task-relationship dilemma’36 37 between the need to provide honest feedback on the task without damaging the relationship between teacher and learner. The studies included in our review suggest that greater attention is being paid to this, as evidenced by validation of measures for the assessment of perceived psychological safety18 and in the debriefing and evaluation of satisfaction.14 23 26 27 The use of video as part of the debrief is not supported by studies included in our review; this is consistent with an earlier meta-analysis.1

Training of debriefers

The majority of studies used trained debrief facilitators to conduct the debrief, although two studies showed that self-debrief within teams was as effective as instructor-led debrief.20 32 Cheng and colleagues,1 in their systematic review of debriefing features, outcomes and effectiveness, found that there may be benefits in expert modelling, although meta-analysis of relevant studies revealed non-significant effects.

When instructors perform debriefs, insimulation debriefing does not work as well as postsimulation debriefing.27 A study examining student perceptions of debriefing38 also revealed that students prefer debriefing immediately following the simulation and that timing was more important than the debriefing model. However, comparison of studies by Cheng and colleagues1 suggest that factors such as task complexity and individual or team-based learning may be better indicators for the timing of debriefing. Further training in specific techniques such as GTSC and CDR raises the quality of debriefings, so it is important to use experienced facilitators, an agreed/previously validated debriefing framework and to supplement facilitator training with technique-specific instruction to optimise debriefing quality. Standards of best practice for simulation39 advocate that the debrief facilitator has specific training and has witnessed the simulation activity. Debriefing frameworks encourage facilitators to focus on a few critical issues, include a range of formats and address technical and cognitive aspects, non-technical skills and transfer of learning into practice.

Quality metrics

We identified four previously validated metrics used to measure the quality of debriefs: DASH, OSAD, REFLECT and DES, with DASH and OSAD the preferred metric in more than one study. These metrics use faculty, participant or objective raters to score aspects of faculty performance except the DES, which assesses participant feelings as a result of the debriefing experience. While these instruments have good evidence of reliability and validity, further studies are needed to establish validity in different contexts and compare the utility of different tools.

Integration with previous work

Previous systematic reviews have shed light on the advantages of debrief over no debrief and the lack of evidence that the use of video improves the debrief.1 11 Our review supports both of these findings. Methods of debriefing have been reviewed in previous narrative reviews2 38 and systematic reviews.1 11 Of note, Cheng and colleagues1 were only able to conduct meta-analysis on a small number of the 177 studies included in their systematic review, due to incomplete reporting by researchers. In a more theoretical approach, the defining attributes of debriefing identified by Dreifuerst10reflection, emotion, reception, and integration and assimilation10—enabled the author to identify model, borderline and contrary cases, in line with the concept analysis method.40

The main contribution of this systematic review has been to identify debriefing frameworks some of which have been validated in various contexts using theoretical approaches. However, the number of bespoke frameworks used highlights the diversity of debriefing practice and approaches to outcome measurement and that more work should be done to compare debriefing frameworks in order to develop evidence for best practice.

Implications for current practice and future research

Our review suggests that the use of a debrief framework improves debrief quality, subsequent behaviours and teamwork performance. The findings strongly support the use of a validated debrief framework by debriefers, but investment in preparation of the faculty is also important to supplement facilitator training with technique-specific instruction to optimise debriefing quality. Further research is needed to validate measures of debrief quality in different contexts and outcome measures following debriefing. The number of bespoke instruments used across the studies illustrates the difficulty with conducting reviews such as this, particularly with limitations to meta-analysis. It would be worth considering whether there are key outcomes (and associated outcome measures) that should be considered good practice for simulation research, similar to the core outcomes dataset approach being promulgated for clinical research (http://www.comet-initiative.org/).

Some key aspects of debrief for team-based simulation, such as facilitator training, the inclusion of a reaction phase and the impact of learner characteristics on debrief outcomes, have no or limited evidence and provide opportunities for future research, particularly with interprofessional groups.

References

View Abstract

Footnotes

  • Contributors All authors fulfil the criteria for authorship; no one who fulfills the criteria for authorship has been excluded. Contributions were as follows: study planning (TG, RE and AO), study conduct (all authors) and development of the manuscript (all authors).

  • Funding This work was supported by the UK Higher Education Authority Teaching Development Grant number GEN-620.

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.