Omodei 11 Omodei and McLennan8 Elliot et al 9 Solodilova-Whiteley and Johnson12
Clear statement of aimsYes. Determine cognitive processes associated with better vs poorer decision making in firefighters.Yes. To compare two methods for studying NDM during orienteering.Yes. To compare two methods for studying NDM in air defence simulations.Yes. To uncover how pilots gather and use information from their environment.
Qualitative methodology appropriate?Yes. Mixed methods.Yes. Mixed methods.Yes. Mixed methods.Yes.
Research design appropriate?Yes, but minimal description.Yes. Participants blinded to study aims.Yes. Three methods of data collection to triangulate results.Yes.
Recruitment strategy appropriate?Unclear. No information provided.Yes. Participants identified by national coaching director of orienteering.Unclear. No information provided.Unclear. No information provided.
Qualitative data collection appropriate?Yes.Borderline. Administration of structured interview protocol: narrative not provided.Yes. Informal observations of researchers.Yes.
Relationship between researcher and participants considered?No. No critical examination of researchers’ roles.Yes.No. No critical examination of researchers’ roles.No. No critical examination of researchers’ roles.
Ethical issues considered?Unclear. No information provided.Unclear. No information provided.Yes. Written consent obtained.Unclear. No information provided.
Rigorous data analysis?Yes. Transcription and a priori coding of debrief commentary with thematic analysis.Borderline. Analysis of semistructured interviews not described.In terms of qualitative component, yes.Yes. Transcription and coding of debrief commentaries. Indepth description of analysis: novel evolutionary technique.
Quantitative componentNo tool described for assessment of performance although inter-rater reliability was good (r=0.81, p<0.1).
Quantitative count of codes: χ2 analysis for differences between groups.
Orienteering checklist:
Wilcoxon rank tests to demonstrate differences in scores between groups.
1. MacShapa program. 2.Transcription and coding with count of recollections: results demonstrated graphically. No specific numerical values (only range), no formal statistical analysis.NA.
Clear statement of findings?Yes. Effective decision making associated with greater emotional self-regulation and monitoring.Yes. CRD superior in terms of eliciting thought processes.Yes. CRD and cognitive walkthrough both effective. Walkthrough more time-efficient.Yes. CRD provided insight into cognitive processes. Development of a cognitive model.
Value of researchYes. Implications for training firefighters.Yes. Discusses future applications in sports research and healthcare.Implications for how we use information and design of human interface systems.Yes. Findings will go on to inform the design of a more user-friendly cockpit interface for pilots.
  • The Critical Appraisal Skills Programme tool was used to assess the quality of the papers included.

  • For mixed-methods papers, an extra column addressed quantitative aspects of the research. Colour coding was applied as a quality indicator—green: no concerns; yellow: minor concerns; red: major concerns.

  • NA, not applicable.