Article Text

Download PDFPDF
Reliability of assessment of medical students’ non-technical skills using a behavioural marker system: does clinical experience matter?
  1. Benjamin Clarke1,2,
  2. Samantha E Smith2,
  3. Emma Claire Phillips1,3,
  4. Ailsa Hamilton1,
  5. Joanne Kerins4,
  6. Victoria R Tallentire1,2
  1. 1 NHS Lothian, Edinburgh, UK
  2. 2 The University of Edinburgh, Edinburgh, UK
  3. 3 Scottish Centre for Simulation and Clinical Human Factors, Scottish Centre for Simulation and Clinical Human Factors, Larbert, UK
  4. 4 NHS Greater Glasgow and Clyde, Glasgow, UK
  1. Correspondence to Benjamin Clarke, NHS Lothian, Edinburgh EH16 4SA, UK; b.clarke{at}doctors.org.uk

Abstract

Introduction Non-technical skills are recognised to play an integral part in safe and effective patient care. Medi-StuNTS (Medical Students’ Non-Technical Skills) is a behavioural marker system developed to enable assessment of medical students’ non-technical skills. This study aimed to assess whether newly trained raters with high levels of clinical experience could achieve reliability coefficients of >0.7 and to compare differences in inter-rater reliability of raters with varying clinical experience.

Methods Forty-four raters attended a workshop on Medi-StuNTS before independently rating three videos of medical students participating in immersive simulation scenarios. Data were grouped by raters’ levels of clinical experience. Inter-rater reliability was assessed by calculating intraclass correlation coefficients (ICC).

Results Eleven raters with more than 10 years of clinical experience achieved single-measure ICC of 0.37 and average-measures ICC of 0.87. Fourteen raters with more than or equal to 5 years and less than 10 years of clinical experience achieved single-measure ICC of 0.09 and average-measures ICC of 0.59. Nineteen raters with less than 5 years of clinical experience achieved single-measure ICC of 0.09 and average-measures ICC 0.65.

Conclusions Using 11 newly trained raters with high levels of clinical experience produced highly reliable ratings that surpassed the prespecified inter-rater reliability standard; however, a single rater from this group would not achieve sufficiently reliable ratings. This is consistent with previous studies using other medical behavioural marker systems. This study demonstrated a decrease in inter-rater reliability of raters with lower levels of clinical experience, suggesting caution when using this population as raters for assessment of non-technical skills.

  • Non-Technical Skills
  • Medical Student
  • Simulation Based Education
  • Reliability

Statistics from Altmetric.com

Footnotes

  • Twitter Emma Phillips @emma_c_phillips.

  • Contributors BC, VRT: Conception of the work; design of the work; data collection, analysis and interpretation; drafting of the paper; revisions to the paper; approved final version of manuscript for submission. SES: Conception of the work; design of the work; data analysis and interpretation; drafting of the paper; revisions to the paper; approved final version of manuscript for submission. ECP: Conception of the work; design of the work; data collection; drafting of the paper; revisions to the paper; approved final version of manuscript for submission. AH and JK: Conception of the work; design of the work; data collection; drafting of the paper; approved final version of manuscript for submission.. All authors agree to be accountable for all aspects of the work.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data availability statement No data are available.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.