- Melinda Fleming1,
- http://orcid.org/0000-0001-5278-3711Michael McMullen1,
- Theresa Beesley2,3,
- Rylan Egan2,4,
- Sean Field2,5
- 1 Department of Anesthesiology and Perioperative Medicine, Queen’s University, Kingston, Ontario, Canada
- 2 Office of Professional Development & Educational Scholarship, Faculty of Health Sciences, Queen’s University, Kingston, Ontario, Canada
- 3 Faculty of Health Sciences, Queen’s University, Kingston, Ontario, Canada
- 4 Health Quality Programs, Queen’s University, Kingston, Ontario, Canada
- 5 Department of Social Anthropology, University of Saint Andrews, Saint Andrews, UK
- Correspondence to Dr Michael McMullen, Department of Anesthesiology and Perioperative Medicine, Queen’s University, Kingston, ON K7L 3N6, Canada;
Introduction Simulation training in anaesthesiology bridges the gap between theory and practice by allowing trainees to engage in high-stakes clinical training without jeopardising patient safety. However, implementing simulation-based assessments within an academic programme is highly resource intensive, and the optimal number of scenarios and faculty required for accurate competency-based assessment remains to be determined. Using a generalisability study methodology, we examine the structure of simulation-based assessment in regard to the minimal number of scenarios and faculty assessors required for optimal competency-based assessments.
Methods Seventeen anaesthesiology residents each performed four simulations which were assessed by two expert raters. Generalisability analysis (G-analysis) was used to estimate the extent of variance attributable to (1) the scenarios, (2) the assessors and (3) the participants. The D-coefficient and the G-coefficient were used to determine accuracy targets and to predict the impact of adjusting the number of scenarios or faculty assessors.
Results We showed that multivariate G-analysis can be used to estimate the number of simulations and raters required to optimise assessment. In this study, the optimal balance was obtained when four scenarios were assessed by two simulation experts.
Conclusion Simulation-based assessment is becoming an increasingly important tool for assessing the competency of medical residents in conjunction with other assessment methods. G-analysis can be used to assist in planning for optimal resource use and cost-efficacy.
- simulation-based education
- curriculum design
Statistics from Altmetric.com
Contributors MF: study conception, interpretation of the data, drafting and revision of the manuscript, and approval of the final version for submission. MM: study conception, interpretation of the data, drafting and revision of the manuscript, approval of the final version for submission. TB: study conception, interpretation of the data, revision of the manuscript and approval of the final version for submission. RE: data analysis and interpretation, revision of the manuscript and approval of the final version for submission. SF: study conception, analysis and interpretation of the data, drafting and revision of the manuscript, and approval of the final version for submission. All authors made important intellectual contributions and can attest to the authenticity of this work.
Funding Queen’s University, Faculty of Health Sciences, Post-graduate Medical Education Special Purpose Grant.
Competing interests None declared.
Ethics approval The study protocol was reviewed and approved by the Queen’s University Health Sciences and Affiliated Teaching Hospitals Research Ethics Board.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available upon reasonable request. All data relevant to the study are included in the article or uploaded as supplementary information.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.