Article Text

Download PDFPDF

Comparison of oral and video debriefing and its effect on knowledge acquisition following simulation-based learning
  1. Omer Farooq,
  2. Victoria A Thorley-Dickinson,
  3. Peter Dieckmann,
  4. Eirini V Kasfiki,
  5. Rasha M I A Omer,
  6. Makani Purva
  1. Hull Institute of Learning and Simulation, Hull Royal Infirmary, Hull, UK
  1. Correspondence to Dr Omer Farooq, Hull Institute of Learning and Simulation, Hull Royal Infirmary, Hull HU3 2JZ, UK; omerfarooqkh{at}gmail.com

Abstract

Introduction Learning from simulation takes place to a large extent in debriefing; however, there is still sparse knowledge about the advantages and disadvantages of debriefing methods. 2 common forms are video-assisted and oral only debriefing. We set out to determine if there is increased knowledge acquisition for candidates undergoing video, rather than oral debriefing.

Methods Second year doctors were randomised to a training day with oral or video feedback. Candidates underwent an emergency clinical scenario followed by feedback. The candidates completed 30 multiple choice questions related to managing common clinical emergencies presimulation and postsimulation training.

Results There was no significant difference in the postscore mean for the oral debriefing (128.3, SD 5.16) and video-assisted debriefing group (127.1, SD=5.96). There was significant interaction between precourse and postcourse scores with superior knowledge acquisition for candidates with lower precourse scores (p=0.008). The candidates with lowest precourse score showed significant improvement in the orally debriefed group in comparison to video debriefed group.

Discussion Our results showed that video debriefing has equivocal effect to oral debriefing. Oral debriefing provides superior knowledge acquisition to learners with lower precourse test scores.

  • Video assisted debriefing
  • Oral Debriefing
  • Randomised
  • Simulation training
View Full Text

Statistics from Altmetric.com

 A recent systematic review concluded that although video-assisted debriefing (VAD) has negligible benefit when compared with oral debriefing (OD), methods related to debriefing and debriefers characteristics were poorly defined in the majority of studies which may have contributed to findings in relation to VAD. Our project incorporated clear debriefer characteristics and a scripted debriefing method to better explore if there was a benefit with VAD. Our study did not demonstrate a benefit for learners with VAD but that OD provided a statistically significant benefit to candidates with lower precourse test scores. Although this study cannot claim to settle the debate on VAD versus OD, we gathered some valuable findings that would allow us to understand the differential effects that VAD might have on learners of different knowledge levels.

Introduction

Debriefing is widely acknowledged as an important factor for facilitating effective learning in simulation.1 Video-assisted debriefing (VAD) uses video playback of a live simulation to provide a record of events. It has been suggested that video playback can add benefit to the learning process by stimulating an outside perspective on the events during the scenario.2 Evidence suggests that the amount of mental effort a learner puts into a lesson influences the amount of knowledge they gain from the lesson.3 Thus, the more they engage in cognitive activities such as perpetual processing, searching memory for cues and elaboration, that is, making connections between old and new information, the more likely they are to gain knowledge. VAD plays back crucial scenario steps, provides visual cues and highlights knowledge gaps more effectively causing an increase in the mental efforts of the learners and higher acquisition of knowledge. Further, 63% of learners4 are visual and auditory learners and are more likely to engage in cognitive activities, which are presented through video format. Hence, we believe that VAD could prove to be more effective than OD in knowledge acquisition.

Although VAD is widely used, there is insufficient evidence to support its use over a purely oral debriefing (OD) model.5 There remain only a small number of studies that have compared VAD to other methods with no overall consensus. Surgical residents' trauma resuscitation skills have been shown to have sustained improvement with a combination of verbal feedback and video playback, while verbal feedback alone was not related to changes in behaviours;6 however, other studies comparing OD with VAD in medical training have found no difference in performance between the two groups.7–9 Further studies have found that despite demonstrating no overall superiority, the OD and VAD methods have shown to be superior in separate learning outcomes like team communication, although in these studies there is no emerging pattern in their individual strengths.10–12 It was suggested that time is an important factor in VAD.2 ,5 ,7 ,8 ,10 ,11 If VAD and OD are given equal time, then playback of video will decrease the total time of discussion during debriefing (as typically, when the video is shown, the discussions stop). So, there might be less time to discuss medical knowledge-related issues.

A recent systematic review13 concluded that although VAD has negligible benefit when compared with non-video debriefing, methods related to debriefing, and debriefers’ characteristics were poorly defined in various studies. Fewer than 3% studies analysed by Cheng et al13 described all seven characteristics of debriefing which are duration of debriefing; educator presence; educator characteristics; content of debriefing; timing; structure and method of debriefing. Hence, there is a need for further studies to explore the differences between OD and VAD, as well as exploration of impact of scripted debriefing on learners is required.

Hence, we designed a study addressing the seven characteristics of debriefing and using a semiscripted debriefing method, to explore if VAD resulted in increased knowledge acquisition compared with OD in a defined group of doctors during a study day addressing management of acute emergencies.

Methods

Study design

The study was a preintervention/postintervention study comparing a group that receives VAD with a group that receives OD. The Sim-Population, Intervention, Comparator, Outcome5 ,8 diagram (figure 1) illustrates the study design. Various aspects of the project are explained below.

Figure 1

Sim-Population, Intervention, Comparator, Outcome diagram.

Study approval

The study was submitted and approved through Integrated Research Application System (IRAS). After review by the local research and development (R&D) department, it was deemed unnecessary to obtain Research Ethics Committee approval, as there were no patients involved. Local R&D approval was obtained. Participation in the study was voluntary. To participate in the study, candidates were given information about the study and had to sign a consent form.

Participants

Candidates were recruited from a regional simulation course for foundation year 2 (FY2) doctors in the UK. FY2 doctors are in their second year of qualified practice and are engaged on a generic training programme. Each FY2 doctor undergoes different rotations, so although their clinical experience varies to some extent they are required to meet the same generic learning outcomes as detailed in the Foundation Program Curriculum.14 In this regard, working with these participants allowed for a compromise of having clinical experience on the one hand and being a somewhat homogeneous participant group, increasing the controllability in our study.

Procedure

An information pack about the study was sent to all potential candidates before the start of the training day and a co-investigator obtained consent on the day. The size of the training groups ranged from 8 to 13 candidates per day.

Candidates undertook a precourse test comprised of multiple choice questions (MCQs; see online supplementary appendix: MCQs) on the morning of the course before the start of simulation scenarios and a postcourse test (see online supplementary appendix). MCQs were completed after the conclusion of the last scenario debriefing. The questionnaires were the same and comprised the identical sets of MCQs on the medical aspects of the scenarios. Candidates were assigned a unique number to record on their precourse and postcourse MCQs. This allowed the scores to be matched while maintaining anonymity.

The training Day

The study encompassed 14 training days that, after randomisation by use of a random number generator, included 6 VAD and 8 OD days. The investigators and authors of the study had no influence on which day a candidate attended, as they were booked by the foundation school, and no one apart from the co-investigators was aware of the style of debriefing to be used prior to the day of the course.

The course was designed to match learning outcomes as set out in the curriculum using a wide variety of settings for the scenarios to reflect the variation in clinical background of the trainees. The theme of the day was ‘management of the acutely unwell patient’. In line with the philosophy of simulation-based education, participants could take an active role in situations that might be too challenging in principle for them in real life (see box 1 for an overview of the scenarios used).

Box 1

Learning objectives from each scenario for the training day

Learning objectives

Scenario 1: sepsis

  • Rapidly makes a focused clinical assessment using ABCDE approach;

  • Recognises and manages the patient with sepsis;

  • Assesses and manages the unconscious patient;

  • Addresses and manages acute pain;

  • To be able to understand initial investigations and apply a prompt and appropriate management plan;

  • To show leadership in the approach to, and management of, sepsis and recognise when to involve senior help;

  • To demonstrate effective communication skills, for example, Situation, Background, Assessment, Recommendation.

Scenario 2: asthma

  • Rapidly makes a focused clinical assessment using ABCDE approach;

  • Recognises and treats asthma;

  • Make appropriate referral within the hospital;

  • Practices in accordance with the principles of confidentiality;

  • Demonstrates extended leadership role within the team by making decisions and dealing with complex situations.

Scenario 3: unconscious end of life patient

  • Rapidly makes a focused clinical assessment using ABCDE approach;

  • Make appropriate differential diagnosis and plan appropriate strategy for further investigation need;

  • Recognises and anticipates deterioration in vital signs;

  • Demonstrates extended leadership role within the team by making decisions regarding escalation of treatment;

  • Demonstrate ability to act on patients' best interest for the patient who lacks capacity;

  • Demonstrates communication skills and empathy when communicating the situation to the family;

  • Safely prescribe appropriate medications, including oxygen to keep the patient as comfortable as possible.

Scenario 4: ruptured abdominal aortic aneurysm

  • Rapidly makes a focused clinical assessment using ABCDE approach;

  • Make appropriate differential diagnosis and plan appropriate strategy for investigation and management;

  • Recognises diagnosis of ruptured Abdominal aortic aneurysm;

  • Anticipates and plans appropriate action to prevent deterioration in vital signs;

  • Demonstrates extended leadership role within the team by making decisions and dealing with complex situations;

  • Makes appropriate referrals within the hospital;

  • Recognise shock or impending shock and resuscitate rapidly;

  • Safely prescribe drugs including reversal of warfarin and observe safe practices in the prescription of blood products.

Scenario 5: status epilepticus

  • Rapidly makes a focused clinical assessment using ABCDE approach;

  • Recognise and treats status epilepticus;

  • Understands appropriate investigations for seizure;

  • Understands treatment for alcohol withdrawal;

  • Make appropriate referral within the hospital;

  • Demonstrates extended leadership role within the team by making decisions and dealing with complex situations.

Scenario 6: cardiac arrest

  • Rapidly makes a focused clinical assessment using ABCDE approach;

  • Demonstrate knowledge about major postoperative complications that can give rise to acutely deteriorating patient;

  • Recognise when to escalate patient and ask for senior help;

  • Be able to provide basic life support until further help comes;

  • Be able to lead a cardiac arrest call, or be part of a team in a cardiac arrest situation following the Advanced Life support protocol;

  • Able to use the defibrillator appropriately.

A training day consisted of an introduction, followed by six scenarios each lasting 15 min being followed by 30 min of debriefing. There were no lectures focused purely on knowledge delivery. At the end of the day, there was a short summarising discussion.

Consistency of the training day was maintained by using the same six scenarios for each training day, a scripted briefing for each scenario, and a semiscripted debriefing (box 2) using the ‘debriefing with good judgement’ model15 delivered by one of three co-investigators. The three co-investigators who conducted the debriefing were postgraduate doctors with at least 4 years postqualification experience and were proficient in the ‘debriefing with good judgement’ model having attended focused training in this technique. The co-investigators were present when the fellow investigator conducted the debriefing and at the conclusion of each day, debriefing of the debriefers took place and actions were agreed to maintain the homogeneity of the debriefing technique.

Box 2

Semiscripted debriefing template

Debriefing tool

Sepsis

Debriefing should focus on the objectives for the scenario, while keeping more general aspects in mind as well.

Knowledge and technical aspects:

  • Criteria for diagnosis of sepsis;

  • Appropriate choice of antibiotics;

  • Components of surviving sepsis bundle;

  • Appropriate investigations;

  • Diagnostic criteria for appendicitis.

Non-technical skills:

  • Systematic approach;

  • Situational awareness;

  • Empathy;

  • Communication—Situation, Background, Assessment, Recommendation;

  • Leadership;

  • Team working.

Semiscripted debriefing guide

We developed a debriefing guide to ensure that the objectives of each scenario were emphasised and uniformity was maintained. This helped us limit the variability in delivery of debriefing. The script focused on technical and non-technical skills. Box 2 provides an example of the script for one of the scenarios used during the project.

Every effort was made to quality assure the debriefing process. The scenarios lasted for 15 min and debriefing was conducted for 30 min to keep the ratio around 1:2. Debriefing was provided immediately after the conclusion of the scenario and was instructor led with inputs from the candidate and the whole group. During the VAD sessions, relevant sections of the video were played back for discussion. Sections were chosen in accordance with both the technical and non-technical learning outcomes set for the scenario.16 OD consisted of the facilitator leading the group through the scenario in chronological order, prompting or facilitating discussion at relevant points chosen in a similar way as for the VAD. Both the VAD and OD structure included the same outcome objectives addressing the medical management of patients and pertinent crisis resource management skills (see box 1).

The knowledge test

Acquisition of knowledge was tested using identical precourse and postcourse (see online supplementary appendix) MCQs. There were 30 stems with five questions in each stem. Negative marking was not used. Therefore, the minimum and maximum score was 0 and 150, respectively. There were five MCQ stems themed for each of the six scenarios, for example, asthma scenario had five MCQ stems around the management for exacerbation of asthma.

MCQs test: reliability and validity

MCQs (see online supplementary appendix) were designed using a robust process, which ensured its validity. They were designed to test factual knowledge, defined and concrete concepts, as well as chains and algorithms, accurately mapped to the generic foundation curriculum and the answers were based on widely accepted local and national guidelines. We consulted two experienced question writers at our local medical school ensuring that the structure and format17 of the questions were appropriate to the objectives of the project and the training day. Further, we ensured that theme of the MCQs and the focus of the training day, matched exactly. They both addressed management of acutely unwell patients in six key conditions. Evidence suggests that the reliability of MCQs test improves as the number of questions increases,18 by including more items in the test for each condition, we believe we further increased the reliability of the test. We contacted consultants with experience of postgraduate examination to ensure consistency17 with similar examinations. Importantly, all candidates were already familiar17 with the question format, as it resembled existing undergraduate and postgraduate examination style.

To ensure face validity, we tested the MCQs prior to roll out using six similar grade doctors for feedback on difficulty, readability, clarity, ‘fairness’ and duration of examination. All the necessary recommendations were incorporated and all parties concerned agreed with the relevance and representation17 of the issues that needed to be discussed on the training day. Once we finalised the MCQs, construct validity was sought and ensured by simultaneous distribution of the proposed test to different levels of doctors. The results validated the final version of the MCQ (see online supplementary appendix) test, as senior doctors did substantially better as compared with newly qualified doctors.

Measures

Candidate MCQ scores immediately after the debriefing were treated as the primary outcome measures.

The secondary outcome measures was the interaction between predicted precourse scores and group to see whether the OD versus VAD scores varies depending on the predicted precourse scores

Sample size calculation

It was planned to use 15 training days each randomly assigned to either OD or VAD (7 of the first and 8 of the second). It was considered important to detect a mean difference of 20% points on the test scores between the VAD and OD groups. Within-group SD of around 10% points would lead to an effect size of 2. An anticipated total of 150 trainees allocated over 15 training days (ie, 10 trainees on each day) allowed detection of an effect size of 0.50 with 80% power using a 5% significance level, assuming an intracluster correlation of 0.15.19

Statistical analysis

A multilevel model20 was to take account of the clusters, that is, training days on which candidates took part. Training days were included in the model as a random effect, with candidates nested within training days. Candidate test scores immediately after debriefing were treated as the outcome measure and the precourse score was included in the statistical model as a covariate. Group differences were tested using a t-test in the multilevel model.

An interaction21 between predicted precourse score for a typical day and group was added to the statistical model to determine whether the OD versus VAD difference varies depending on the predicted precourse score.

Results

Data were collected from a total of 139 candidates. Sixteen candidates were excluded from the data analysis as they did not consent to be part of study but they participated in the day's proceedings. There were 75 candidates in OD group and 64 in VAD group.

Overall the postcourse score means for the OD group and VAD group were 128.3 (SD=5.16) and 127.1 (SD=5.96), respectively (see table 1). The between-training day variance was estimated to be 0.449 and the within-training day variance was estimated to be 20.01, leading to an intra-training day correlation of 0.022. The difference in test scores overall between the OD and VAD groups was not statistically significant (t=2.07, df=12, p=0.06).

Table 1

Scores for each training day and debriefing style assigned

There was a significant interaction between predicted precourse scores and group (estimate=0.275, SE=0.102, t=2.69, df=123, p=0.008; table 2). If the precourse score was 105, then OD was predicted to add around 6 points more (124.5–118.4) to the postcourse score. However, if the precourse score was 125 then the predicted extra was less than a 1-point improvement (130.1–129.5).

Table 2

Predicted postcourse scores for groups based on precourse scores for video-assisted debriefing and oral debriefing candidates

Discussion

Our study demonstrated that OD provided a statistically significant benefit over VAD to candidates with lower precourse test scores. However, we were unable to demonstrate an overall difference in knowledge acquisition as measured by change in MCQ test scores between the OD and the VAD groups.

Our results (table 2) indicate that the lower the predicted pretest score, the greater the benefit for both groups; however, the lower scoring OD candidates had a statistically significant additional 6 points benefit over the lower scoring VAD candidates. Conversely, the data showed that candidates with high precourse scores had the smallest improvement in their scores, with no significant difference between the OD and VAD groups—this could possibly be explained by a ceiling effect. There was a marked intragroup variability in pretest score, with little intergroup variability, showing overall that individual candidates were heterogeneous, despite the fact that they were at the same stage of their training. This highlights that candidates come to training with a range of personal learning needs, and that the method of delivery we choose to use may not be of equal benefit to all learner groups. This view point is echoed in the recent meta-analysis by Cheng et al,13 who suggested that current study findings for VAD may be learner group specific and further studies are required to examine content, groups or environments where debriefing takes place.

When Chronister and Brown,11 compared OD with VAD for nursing students, they found that the candidates who underwent OD alone showed a superior improvement in knowledge, while the VAD group had a larger improvement in skills. Our findings would echo this improvement in knowledge from OD but we did not follow-up delegates to address the improvement in skills. In a similar study by Reed et al10 with nursing students, three areas of significant difference were identified in self-reported postcourse ratings.

By default, the replay of the video possibly sets the focus more on the behavioural aspects of the scenario—as, by its nature, the video shows an outside perspective. The video playback time also adds inconsistency to the length of debriefing as the amount of time spent replaying the video may differ between facilitators and scenarios. This also varied from study to study in the literature. In Sawyer et al's8 study the entire scenario was replayed; however, in other studies individual clips were selected. We used clips with bookmarking to highlight various aspects and where there were ambiguities about the candidates' response, video replay was used for clarification. As there is little guidance on the use of video playback in debriefing it is difficult to know how to adjust for this; however, future studies should take this into account.

Previous studies have highlighted reasons why VAD may be less effective for learners with knowledge gaps. Savoldelli et al7 reported that video playback could be distracting, as combined verbal and video feedback could lead to ‘information overload’. On the other hand, many studies also described VAD as engaging, and reported high candidate satisfaction or preference for this method.2 ,6 ,9 ,12 ,22 Our candidates did not experience both styles of debriefing, so we are unable to comment on preference. It seems that the amount of time spent on video playback, and whether that time spent is engaging or distracting, largely rests on the abilities of the debriefer2 and supports the need for skilled facilitators with experience of the specific technique used.23

We could alternatively ask why VAD did not produce better MCQ scores despite the reported preference by candidates, facilitators and evidence base in areas outside medicine.8 Reported benefits of VAD include identifying discrepancy in perceived self-efficacy,6 allowing learners to see themselves from a new perspective24 and offering an accurate account of events in a non-threatening way.12 Although these are all valid advantages, there is no obvious connection in how this would improve clinical knowledge, and therefore it would be unlikely that this would be reflected in our study results. We believe that inconsistent and varied results of studies to date are a reflection of the complexities of simulation-based education and debriefing. Debriefing is guided by the learning outcomes set for a particular scenario, which may be cognitive, technical or behavioural or any combination of those.24 The debriefing process contributes to the variation in the learning process as it is influenced by who debriefs whom and how the dynamics between those involved evolves. Further adding to the complexity are the candidates and their response to the debriefing process. The candidates' mental journey during the debriefing is influenced by their pre-existing knowledge base and prior (clinical) experiences,15 ,23 besides the actual dynamics in the debriefing. Also the method used to measure the impact, that is, MCQs or repeat scenarios would add to variance in results. Hence, it is difficult to compare two different styles of debriefing as standardisation of the process and reducing the variables in this process may be very difficult to achieve.

Our study has demonstrated that OD provided a statistically significant benefit over VAD to candidates with lower precourse test scores, when considering the measuring of knowledge gained. Although this study cannot claim to settle the debate on VAD versus OD, we have focused on addressing all the aspects of debriefing and thus investigating the dynamics behind the score in more detail.

Further on the study addresses the key issue highlighted by Cheng et al13 that very few studies highlight debriefing methods in detail. We have used MCQs (see online supplementary appendix), which focus on the knowledge acquisition of an individual, and semiscripted debriefing which highlights technical and non-technical aspects in greater depths. Our study adds valuable evidence to knowledge base to help guide facilitators and future study.

Limitations of our study

The limitations of our study lie in the use of the identical questions for pretests and post-tests which may have led to overestimation of the participants' knowledge in the post-test. One could further argue that teaching cognitive skills alone may be better achieved using low-cost methods such as self-study. Simulation, especially VAD, is meant to set the focus on behavioural aspects, likely also in our study. Yet, we focused on the cognitive aspects in the investigation of the effect. The scope of the project did not allow doing a proper analysis of behavioural changes. Yet, the differential effects on knowledge acquisition in our viewpoint highlights the need for further individualise simulation-based learning and to find out more about the differences between participants that are relevant in this context.

Certainly much more is happening during a debriefing and to account for the complexity of the learning process, just measuring knowledge in MCQ certainly does not give a complete picture. Our study is an indication for the connection between the learning benefits of participants, the debriefing method used and the learning goals achieved.

We tried to limit the amount of variation in the participants by focussing on a specific group of participants. However, what seems to be a homogeneous group actually turned out to be people with rather different backgrounds. In future studies varying the participants' characteristics systematically will help in identifying those characteristics that make a difference for the learning success.

Conclusion

In this study, we found differential effects of the use of video during debriefings on the knowledge acquisition of participants. Persons with low precourse test scores improved their knowledge more in the OD group.

References

View Abstract

Footnotes

  • Competing interests None declared.

  • Ethics approval Research and Development Department, Hull and East Yorkshire Hospitals NHS Trust.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.