Article Text


Building consensus for the future of paediatric simulation: a novel ‘KJ Reverse-Merlin’ methodology
  1. Elizabeth A Hunt1,
  2. Jordan Duval-Arnould1,
  3. Nnenna O Chime1,2,
  4. Marc Auerbach3,
  5. David Kessler4,
  6. Jonathan P Duff5,
  7. Nicole Shilkofski1,6,
  8. Marissa Brett-Fleegler7,
  9. Vinay Nadkarni8,
  10. Adam Cheng9
  11. for the International Network for Simulation-based Pediatric Innovation, Research and Education (INSPIRE) Investigators
    1. 1Johns Hopkins University School of Medicine, Baltimore, Maryland, USA
    2. 2Albert Einstein College of Medicine and Children's Hospital at Montefiore, Bronx, New York, USA
    3. 3Yale University School of Medicine, New Haven, Connecticut, USA
    4. 4Columbia University College of Physicians and Surgeons, New York, New York, USA
    5. 5University of Alberta, Edmonton, Alberta, USA
    6. 6Perdana University Graduate School of Medicine
    7. 7Boston Children's Hospital, Boston, Massachusetts, USA
    8. 8Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, USA
    9. 9Alberta Children's Hospital, Calgary, Alberta, Canada
    1. Correspondence to Dr Nnenna O Chime, Department of Pediatrics, Children's Hospital at Montefiore, 111 East 210th Street, Rosenthal 4, Bronx, NY 10467, USA; nnenna_chime{at}


    Objectives This project aims to identify guiding strategic principles to optimise simulation-based educational impact on learning, patient safety and child health.

    Methods Study participants included 39 simulation experts who used a novel ‘KJ Reverse-Merlin’ consensus process in the systematic identification of barriers to success in simulation, grouped them in themes and subsequently identified solutions for each theme.

    Results 193 unique factors were identified and clustered into 6 affinity groups. 6 key consensus strategies were identified: (1) allocate limited resources by engaging health systems partners to define education and research priorities; (2) conduct and publish rigorous translational and cost-effectiveness research; (3) foster collaborative multidisciplinary research and education networks; (4) design simulation solutions with systems integration and sustainability in mind; (5) leverage partnerships with industry for simulation, medical and educational technology; (6) advocate to engage the education community, research funding agencies and regulatory bodies.

    Conclusions Simulation can be used as a research, quality improvement and or educational tool aimed at improving the quality of care provided to children. However, without organisation, strategy, prioritisation and collaboration, the simulation community runs the risk of wasting resources, duplicating and misdirecting the efforts.

    Statistics from


    Simulation has great potential to improve quality across healthcare and is increasingly being used to educate, facilitate research, and to improve safety, processes and quality of care, and patient outcomes.1–6 The exponential growth of simulation centres throughout the world reflects the degree to which educators, learners and administrative decision makers have embraced this innovation. As the field matures, international professional organisations and networks have emerged such as the International Pediatric Simulation Society (IPSS), the Society for Simulation in Healthcare (SSH), the Society in Europe for Simulation Applied to Medicine (SESAM) and the Network Pediatric Simulation to facilitate knowledge development and dissemination. Within these organisations, clinicians, researchers and educators who use simulation have established specialised networks. Two such paediatric groups developed independently, utilising healthcare simulation to develop educational interventions and perform multicentre collaborative research: Examining Pediatric Resuscitation Education using Simulation and Scripting (EXPRESS)7 and Patient Outcomes In Simulation Education (POISE) networks.8

    Strategic leadership from EXPRESS and POISE organised a working group to discuss the future direction of paediatric simulation-based education (SBE) and paediatric simulation-based research (SBR). The aim of this process was to leverage the collective expertise of its members and build consensus on guiding strategic principles to be considered in order to optimise their impact on learning, patient safety and ultimately on child health.



    Study participants included the leadership and a convenience sample of members from the EXPRESS and POISE networks present at the joint collaborative meetings at the International Meeting for Simulation in Healthcare (IMSH) and the International Pediatric Simulation Symposium and Workshops (IPSSW). Both networks had experts in simulation, paediatric clinical care, research methodology and education theory with approximately 114 members from over 40 institutions across the world at the time of these meetings.9 The primary goal established by the EXPRESS network was to improve paediatric outcomes by performing multicentred research studies related to paediatric resuscitation, while the goal of the POISE network was to develop and disseminate simulation-based educational interventions that result in measurable improvements in paediatric patient outcomes. The leadership of both networks had a secondary goal to train and mentor young investigators, thus nurture tomorrow's leaders in SBE and SBR.

    Target audience

    The target audience for this consensus document includes paediatric clinicians, educators, researchers, hospital leaders, regulatory agencies, policymakers, advocacy groups and funders of healthcare research.

    Consensus building methodology

    Conference organisers reviewed the literature and selected several methods of systematically identifying barriers to success and their associated solutions and mapped out a staged process for developing consensus (see figure 1).

    Figure 1

    Flow diagram of KJ Reverse-Merlin consensus process. EXPRESS, Examining Pediatric Resuscitation Education using Simulation and Scripting; IMSH, International Meeting for Simulation in Healthcare; IPSSW, International Pediatric Simulation Symposium and Workshops; POISE, Patient Outcomes In Simulation Education.

    KJ method (also known as affinity diagram technique)

    The KJ method is a ‘brainstorming technique’ designed in the 1950s by Japanese scientist Jiro Kawakita.9 ,10 Key features include a period where participants independently work in silence, generating ideas in response to a question and then work in parallel attempting to create thematically similar clusters, that is, an affinity diagram.11 This non-verbal approach stimulates the creative right brain and facilitates balanced input from introverted and extroverted participants.12–14

    Merlin exercise

    Merlin was a mythical magician who ‘lived his life backwards’, living in the future, but not knowing what steps were taken to get there from the present state. Like Merlin, participants were asked to envision that their organisation is exceptionally successful in the future, and then work from this future state backwards to imagine and describe what occurred to enable their organisation's success. This process identifies potential changes that need to take place in order to meet defined milestones. Smith describes The Merlin Factor as “the ability to see the potential of the present from the point of view of the future”.15 Examples of successful applications of the Merlin exercise abound in the business literature.16–19

    Adapted KJ Reverse-Merlin exercise

    Unlike a traditional Merlin exercise that asks participants to imagine a future state of success, our KJ Reverse-Merlin process asked participants to imagine a future state of failure. This portion of the process is very similar to a project ‘premortem’ process described by Gary Klein in which he describes the use of ‘prospective hindsight’, another useful technique adapted from business where team members generated ‘plausible reasons for the project's failure’.20 Their responses were intended to help proactively develop strategies to navigate predictable roadblocks.

    ‘Reverse-Merlin’ question

    The ‘Reverse-Merlin’ question posed by the working group leadership was: ‘It is the year 2020, and we have failed to successfully use simulation to improve healthcare education, processes of care and patient outcomes for children—what went wrong?’

    The KJ Reverse-Merlin exercise took place immediately preceding the IMSH in January 2011. Participants were given 15 min to brainstorm in silence and independently identify factors that could plausibly contribute to the proposed future state of failure for paediatric simulation; each discrete factor was written on a ‘sticky note’. They were then directed to randomly stick each note on a large window in the conference room. All participants then simultaneously, but still independently and silently, grouped factors that seemed to be related into clusters (see figure 2). Any member could move any factor into or out of a cluster, move multiple factors to create a new cluster, or replicate a factor such that it could be added to two or more clusters.

    Figure 2

    Photograph of participants during ‘affinity diagram’ clustering process.

    When the key clusters seemed to have been created, the silence was finally broken. At that point, each cluster was identified as an ‘affinity group’ and working group members divided into smaller groups (3–4 members+1 facilitator). Working collaboratively, each affinity group identified the top five barriers and brainstormed possible solutions for each barrier. During the subsequent small group stage, each affinity group created a matrix prioritising the identified factors, clarifying how these factors represented potential barriers to success and listed corresponding possible solutions. At the end of that meeting, each affinity group facilitator presented a summary of their work and received feedback from the entire consensus group.

    A follow-up meeting and discussion was conducted at the face-to-face EXPRESS research meeting following the IPSSW in October 2011. Attendees were given an overview of the process to date and subsequently asked to electronically review the draft affinity group statements and make comments that could be read by other reviewers, and ultimately to give a thumbs up or down vote indicating whether or not they approved with each draft statement.


    There were a total of 39 participants throughout the process from 32 institutions, 9 countries and 5 continents. This represented all of the senior leadership and 34% (39/114) of the combined EXPRESS and POISE membership. The group had a wide range of expertise including diverse paediatric clinical specialties (emergency medicine, critical care, neonatology, anaesthesiology, pain management, general paediatrics and international outreach), educational theory and assessment, psychology, clinical research, epidemiology and informatics.

    Twenty-five members participated during the Reverse-Merlin brainstorming and KJ clustering stages. Twenty-one of the aforementioned 25 members participated in the affinity group discussion stages at IMSH. The Reverse-Merlin exercise yielded 193 unique factors contributing to failure. Of the 193 factors, some were seen to fall under similar constructs. Subsequently, 132 factors were grouped into clusters. During the KJ clustering stage, eight notes were replicated to be included in multiple clusters. At the end of the follow-up discussion, the clusters were ultimately transformed into six affinity groups, that is, all 193 factors were distributed to six affinity groups (see figure 3). Each affinity group used the matrix created as the foundation for subsequent affinity group statements.

    Figure 3

    Factors identified during KJ Reverse-Merlin process that might contribute to future failure.

    Twenty-two members including 8 who had participated in the brainstorming and clustering stages and 14 who had not yet participated in the process participated in the affinity group electronic voting phase held at the IPSSW meeting. There was a 91% (20/22) overall response rate in this electronic phase, that is, participants voting on at least one affinity group statement and an 86% (114/132 possible responses) complete response rate, that is, each participant voting on all of the six affinity group statements. The approval rate of this draft affinity group statements ranged from 78.9% to 94.7%.

    Each affinity group leader summarised key issues raised in each affinity group discussion and provided the first draft of the consensus statement for each group. These drafts were circulated to the consensus working groups for feedback, with edits and revisions incorporated through an iterative process.

    Affinity group reports


    Potential barriers

    In today's economy, the cost of healthcare delivery is receiving ever-increasing scrutiny. Regardless of the enthusiasm of our learners and emerging data on the powerful impact of SBE,21 we cannot expect an unending growth in financial resources to support simulation education and research. In addition, increasing work hour restrictions for physician trainees by key regulatory bodies profoundly limits the amount of time learners are able to spend on educational activities. The available educational time of healthcare providers at the training and continuing education levels are similarly a finite resource. Failure to prioritise how simulation resources, including learner time, are spent will limit its ultimate impact.


    It is imperative that the simulation community focuses the use of simulation technology to arenas where it will be most helpful. The simulation toolbox contains a wide variety of techniques and strategies that must be matched with the spectrum of clinical needs, ranging from the currently emphasised critical event management and procedural skill acquisition,22 to educational objectives as diverse as an understanding of pathophysiology of head injury and the approach to difficult conversations in paediatrics. Engaging healthcare professionals, educators and researchers from across the continuum of the healthcare system is necessary to identify priorities for SBE and SBR moving forward.

    Research methodology and outcomes

    Potential barriers

    Governing and regulatory bodies in paediatrics are mandating the use of simulation for training and assessment.23–25 Existing research has defined best practices for simulation-based interventions with the practitioner serving as the focus of measurement most commonly assessed in a simulation setting.8 ,26 ,27 However, there is a paucity of research linking simulation to improved clinical performance, safe processes of care, patient outcomes or cost-effectiveness.28–32 The field must avoid the temptation to study only questions that can be easily measured, rather than what is truly important.

    Solutions: Researchers must develop the methods and infrastructure to assess the translation between simulation and clinical care.32 Future studies should control for potentially confounding factors in this complex healthcare system that impact patient outcomes, so that we can understand the impact of simulation on patient care. Over time, researchers must develop and rigorously evaluate the most effective components of simulation-based interventions from a patient-centered perspective. Analysis will be conducted to identify the most cost-effective approaches to the deployment of simulation resources and explore issues related to decay rates and retraining intervals. The healthcare system should support simulation with additional funding and resources to the extent that the research community demonstrates that simulation improves educational and clinical outcomes. Funding for this research is essential to the definition of best practices in simulation that improve clinical outcomes and to justify their subsequent funding.

    Academic collaboration

    Potential barriers

    Many SBR studies suffer from small sample sizes, suboptimal design or lack of clinically significant outcome measures.33 ,34 When research networks do exist, physicians are frequently over-represented; we must therefore ensure that these networks are broad enough to include all members of the healthcare team as well as education research experts. Similarly, simulation-based curricula developed at a single institution may lack generalisability to other institutions.


    Paediatric simulation-based researchers and educators must collaborate in multi-institutional efforts, ultimately enabling studies and curricula to be conducted that otherwise would be impossible. For example, even large paediatric simulation programmes struggled to enrol 16 multidisciplinary teams (∼70 individuals) for the EXPRESS trial, but collectively 14 study sites enrolled 104 teams (453 individuals), enabling conduct of a large randomised controlled trial that was otherwise impossible. To facilitate collaborative studies, the EXPRESS (now part of International Network for Simulation-based Pediatric Innovation, Research and Education—INSPIRE) collaborative created a free web-based research portal that facilitates collaboration and sharing of data and videos between multiple recruitment sites.35 Similarly, the POISE network (also now INSPIRE) developed a monthly webinar series focused on the science of SBE and SBR methodologies and has now partnered with IPSS to make this curriculum available to an international audience of paediatric simulation enthusiasts. Assessment tools, and the methodology applied to validating these tools, should be shared among researchers as well.


    Potential barriers

    Systematic integration of simulation across the healthcare enterprise and long-term sustainability has received little consideration. In many cases, lack of institutional support limits SBE programmes in their ability to grow, and implicitly conveys a message that the institution does not recognise a link between the value of SBE and the institution's mission. This may be related to the fact that the impact, cost and return on investment of SBE is not sufficiently defined. If these emerging data are not communicated and applied effectively, the benefits of SBE will not be translated to front-line providers and their patients.


    In order to be sustained, SBE must become an integral part of excellent delivery of healthcare, and no longer regarded as ‘supplemental’. A review and synthesis of 19 empirical health-related studies reports five important factors influencing the extent of sustainability: (1) programme modification, (2) presence of a ‘champion’, (3) ‘fit’ with organisational mission and strategic plan, (4) perceived benefits to staff members and/or clients, and (5) stakeholders provide support.36 In order to gain institutional support from key stakeholders, demonstration of the superior effectiveness of SBE over traditional training methods and active engagement of leadership as champions is necessary. Cost-effectiveness data are needed to provide organisations with relevant information to make financial decisions and allocate appropriate educational resources to SBE. In addition, working directly with institutional leaders to align simulation curricula or research with organisational goals can improve the perceived fit between the two. For example, an annual review of a healthcare system's sentinel events may reveal areas in which hospital leadership would be enthusiastic about supporting simulation training, for example, low-frequency, high-risk events such as deaths related to failed management of difficult airways or maternal haemorrhage. Similarly, targeting known high-frequency but difficult to eradicate problem areas, such as errors in handover and medication reconciliation as well as high hospital readmission rates, is likely to engage senior leadership and ultimately impact a large proportion of patients. Finally, consideration of adult learning theory principles in terms of ensuring that simulation curriculum is timely, easily accessible and directly relevant to the front-line providers is essential.


    Potential barriers

    Simulation is best framed as a teaching and/or research technique and not a technology; however, there are several technology-related issues that deserve attention.1 Existing simulators are not adequate to meet paediatric-specific needs in their realism, variety and cost. They fail to span age, anatomic and physiological ranges found in paediatric patients. In addition, higher technology simulators can be prohibitively expensive, and are often not durable or appropriate for austere environments.

    Another important technology consideration is that simulation centres are uniquely suited for usability testing of medical devices to ensure patient safety. However, incompatibilities between simulators and medical equipment and lack of standards prevent efficient testing and the ultimate goal of seamless and safe integration of new technology into clinical environments.


    Societies and consumer groups should work collectively to address prohibitive costs through shared purchasing models. Research and development should be encouraged by leveraging policymakers to incentivise industry cooperation with academic centres. The development of interdisciplinary and interinstitutional collaboration may optimise cost-benefit ratios.37 Telemedicine/telesimulation programmes hold potential to increase the impact of shared resources. Research identifying the appropriate level of realism for particular applications of SBE also has the potential to help control costs.

    Working with the National Office of Standards and Technology to encourage compatibility between medical devices and healthcare simulation equipment can help drive better systems integration, scalability and realism. It is essential that technology—the simulators themselves, the technology of the healthcare environment and that of the educational experience—be thoughtfully developed to maximise the impact on clinical outcomes.


    Potential barriers

    These include (1) lack of commitment to maintenance of human resources (simulation educators); (2) lack of adequate financial, space and material resources; and (3) underdeveloped relationships between simulation experts, regulatory organisations (Joint Commission, ACGME, etc) and the general public, which might otherwise promote endorsement of simulation.


    Acknowledgement that paediatric simulation programmes are valuable and will not flourish without support is an essential step. Institutions must develop plans to amplify paediatric simulation expertise and then provide faculty development programmes, educator time, space, equipment and administrative support to sustain programmes. Support for paediatric simulation resources requires generating buy-in from medical and scientific communities, and other key stakeholders. This demands effective lobbying on local, regional and national levels. Government leaders must understand that simulation is critical to training providers to manage high-risk, low-frequency events.38 ,39 Ongoing advocacies by paediatric simulation communities will increase public buy-in, leading to appropriate resources to meet mutual goals, and eventually to increasing numbers of educators and researchers receiving grant funding. These individuals can then garner protected time to become leaders within research, education, administration and regulatory bodies.

    Metamorphosis from EXPRESS and POISE into INSPIRE

    After reviewing the findings of the described consensus process, the strategic leadership of EXPRESS and POISE recognised the similarities and compatibility between the two existing paediatric simulation research networks. This process helped to identify redundancies in membership, projects and structure of the two networks. In order to eliminate these redundancies and maximise synergy, the decision was made to merge efforts into INSPIRE, thus combining the structure, mission and vision of both existing networks to optimise efforts to advance the field of paediatric simulation and their impact on child health globally (see figure 4).

    Figure 4

    Map of INSPIRE sites. INSPIRE, International Network of Simulation-based Pediatric Innovation, Research and Education.


    The use of healthcare simulation has great potential to improve the quality of care for children and will ultimately save more lives. However, without organisation, strategy, prioritisation and collaboration, we run the risk of wasting resources, duplicating efforts and misdirecting our efforts. Our consensus summary statement highlights six areas that deserve special consideration for the paediatric simulation community to use as guiding principles for the future. Strategies that we must utilise in order to optimise the impact we make on learning, patient safety and ultimately on child health include (1) make the best use of limited resources by engaging partners across our health systems to define education and research priorities; (2) conduct and publish rigorous translational and cost-effectiveness research; (3) foster collaborative multidisciplinary research and education networks; (4) design simulation solutions with systems integration and sustainability in mind; (5) partner with industry and strategise collectively in order to optimise the impact of simulation, medical and educational technology; and (6) utilise advocacy to engage the education community, research funding agencies and regulatory bodies.


    The authors wish to thank the INSPIRE Network and the INSPIRE Network Executive Committee for providing the infrastructure support for this study and manuscript. The authors also wish to thank the Laerdal Foundation for Acute Care Medicine for funding the INSPIRE network infrastructure as well as the RBaby Foundation for providing support for the Network. In addition, the authors wish to express appreciation to the Society for Simulation in Healthcare and the International Pediatric Simulation Society for supporting the INSPIRE bi-annual meetings that occurred in conjunction with their annual conferences.


    View Abstract


    • Collaborators INSPIRE Investigators include Abdullah Alqahtani (King Abdulaziz Medical City (KAMC), Riyadh, Saudi Arabia,; JoDee Anderson (Oregon Health and Science University, Portland, Oregon, USA,; Alberto J de Armendi (University of Oklahoma Health Sciences Center, Oklahoma City, Oklahoma, USA,; Jennifer Arnold, Texas Children's Hospital, Houston, Texas, USA,; Mary Bennett (British Columbia Children's Center, Vancouver, British Columbia, Canada,; Dylan Bould (Children's Hospital of Eastern Ontario, Ottawa, Ontario, Canada,; Linda Brown (Hasbro Children's Hospital,; Todd P Chang, Children's Hospital Los Angeles, Los Angeles, California, USA,; Walter J Eppich (Ann & Robert H Lurie Children's Hospital of Chicago, Chicago, Illinois, USA,; Tobias Everett (The Hospital for Sick Children, Toronto, Canada,; Marino Festa (The Children's Hospital at Westmead, Westmead, New South Wales, Australia,; Vincent J Grant (Alberta Children's Hospital, University of Calgary, Calgary, Alberta, Canada,; Louis P Halamek (Stanford University School of Medicine, Palo Alto, California, USA,; Laura Haubner (University of South Florida College of Medicine, Tampa, Florida, USA,; Takanari Ikeyama (Tokyo Metropolitan Children's Medical Center, Tokyo, Japan,; Chris Kennedy (Children's Mercy Hospital, Kansas City, Missouri, USA,; Ralph Mackinnon (Royal Manchester Children's Hospital, Manchester, UK,; Renuka Mehta (Georgia Health Science University, Augusta, Georgia, USA,; Elaine Meyer (Boston Children's Hospital, Boston, Massachusetts, USA,; Akira Nishisaki MD (Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, USA,; Frank Overly (Hasbro Children's Hospital, Providence, Rhode Island, USA,; Jennifer Reid (University of Washington School of Medicine, Seattle Children's, Seattle, Washington, USA,; Robert Simon (Center for Medical Simulation—Harvard Medical School, Cambridge, Massachusetts, USA,; David Stewart (Royal Manchester Children's Hospital, Manchester, UK,; Nancy Tofil (University of Alabama at Birmingham, Children's of Alabama, Birmingham, Alabama, USA,; Kimberly Stone (University of Washington School of Medicine, Seattle Children's, Seattle, Washington, USA,; Kathleen Ventre (Children's Hospital Colorado, Aurora, Colorado, USA,; Burkhard J Wermter (MHH Medical School Hannover, Hannover, Germany,; Marjorie Lee White (University of Alabama at Birmingham, Children's of Alabama, Birmingham, Alabama, USA,; Jason Zigmont (Ohio Health Experiential Learning, Columbus, Ohio, USA,

    • Contributors All authors were part of the conceptualisation and design of this study. All authors participated in the consensus exercise, were part of the initial draft, review and approval of the final draft of the manuscript as it is.

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    Request permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.