Article Text


Prospective, randomised and blinded comparison of proficiency-based progression full-physics virtual reality simulator training versus invasive vascular experience for learning carotid artery angiography by very experienced operators
  1. Christopher U Cates1,
  2. Lars Lönn2,
  3. Anthony G Gallagher1,3
  1. 1Division of Cardiology, School of Medicine, Vascular Intervention, Emory University Hospital, Atlanta, Georgia, USA
  2. 2Department of Radiology and Vascular Surgery, National Hospital, Copenhagen University, Copenhagen, Denmark
  3. 3ASSERT Centre, School of Medicine, University College Cork, Cork T12 DX01, Ireland
  1. Correspondence to Professor Anthony G Gallagher, ASSERT Centre, School of Medicine, University College Cork, Room 2.103B School of Medicine, Brookfield Health Sciences Complex, Cork, Ireland; ag.gallagher{at}


Introduction We assessed the transfer of training (ToT) of virtual reality simulation training compared to invasive vascular experience training for carotid artery angiography (CA) for highly experienced interventionists but new to carotid procedures.

Methods Prospective, randomised and blinded.

Setting Catheterisation and skills laboratories in the USA.

Participants Experienced (mean volume=15 000 cases) interventional cardiologists (n=12) were randomised to train on virtual reality (VR) simulation to a quantitatively defined level of proficiency or to a traditional supervised in vivo patient case training.

Outcome measures The observed performance differences in performing a CA between two matched groups were then blindly assessed using predefined metrics of performance.

Results Experienced interventional cardiologists trained on the VR simulator performed significantly better than their equally experienced controls showing a significantly lower rate of objectively assessed intraoperative errors in CA. Performance showed 17–49% ToT from the VR to the in vivo index case.

Discussion This is the first prospective, randomised and blinded clinical study to report that VR simulation training transfers improved procedural skills to clinical performance on live patients for experienced interventionists. This study, for the first time, demonstrates that VR simulation offers a powerful, safe and effective platform for training interventional skills for highly experienced interventionists with the greatest impact on procedural error reduction.

Statistics from


Training in medicine is currently going through a paradigm shift. The provision of one-on-one apprenticeship learning of doctors is abrogated by high-profile error cases1–3 and radical reductions in acceptable duty hours and therefore training experience. The settled solution appears to be the harnessing of technology-enhanced learning, particularly simulation-based training.4 Also recently, the Department of Health in the UK has directed that a procedure should not be performed on a patient, the first time that it is performed.4

Virtual reality (VR) simulation as an approach to training skills was first validated in 2002 for a surgical procedure5 and was subsequently adopted and championed by American Surgery in 2006.6 This method has also been enthusiastically embraced by interventional cardiologists (IC) with the adoption of VR simulation training for the training of new interventional devices.7–9 This is not surprising given the rate of change and evolution of cardiovascular devices and the morbidity associated with some new procedures (eg, transcatheter aortic-valve implantation (TAVI), carotid stenting, stroke intervention), even by experienced operators.10 Furthermore, vascular medicine has access to the most sophisticated VR simulators currently available in healthcare.9

There is a relative lack of published studies evaluating the impact of VR simulation training on actual individual operator clinical performance on patients.9 In a review of simulation studies on intravascular procedures, Desender et al11 reported only two transfer of training (ToT) studies of VR simulation training on in vivo performance.12 ,13 De Ponti et al14 reported the beneficial effects of VR simulation training in comparison to conventional training on transseptal catheterisation performance. Indeed, none of the prospective, randomised clinical validation studies that have evaluated simulation training have determined its utility for training highly experienced operators learning a new technique or new procedural skill, even in surgery.15

The aim of this study is to evaluate the utility of VR simulation training in comparison to one-to-one proctored/mentored in vivo training for highly experienced IC attempting to learn a new procedure, that is, carotid angiography (CA).



Twelve experienced, attending level, IC (ages 34–68 years, mean 48 years), with a mean interventional coronary case volume of 15 000 cases (range 2000–15 000 +), but with no experience in CA.

Randomisation and recruitment

Subject numbers 1–12 were recruited and organised in age/experience matched in t pairs to eliminate age/experience bias. They were assigned to one of two groups (figure 1) using an online research number randomiser ( Their number and group allocation were kept in serial numbered envelopes and retrieved (from an independent third party) just prior to subject participation in the study. All participants underwent basic didactic training on CA technique. All 12 of the participants underwent the Carotid Artery Stenting Education System (or CASES education system, eTrinsic, Inc (Denver, Colorado, USA) and purchased by Simbionix, Cleveland, Ohio, USA) before their first case. CASES is a comprehensive online education solution providing ‘hands-on’ training. The online training includes instruction on the carotid anatomy and aortic arch types and their relevance to procedure performance and the devices to be used for performance of the procedure. It also includes instruction on the steps of the procedure including a simulation of a physical object (catheters, wires, stents (of different lengths) and embolic protection device that can be manipulated by a user (with a computer mouse) and the executable program having a monitoring capability. Independently, we utilised standardised video and knowledge assessments of basic technique for which participants demonstrated excellent didactic performance with passing scores (100%). Participant operators were assigned in matched pairs (based on age and experience) and then one of the pair was randomly allocated to the VR training versus conventional training. Participants were trained by independent researchers on the basis of information retrieved from the sealed envelope. Outcomes assessments and analysis were conducted by individuals blinded as to subject identity and their training group status.

Figure 1

Study design and flow of trainees through the trial.


The Vascular Interventional Simulation Trainer (VIST; figure 2A—D), described elsewhere, was used for the study.16 ,17 The system uses a geometrical vessel representation together with physics-based calculations to determine the behaviour of the interventional devices. The simulation uses programmed properties of the devices (stiffness, friction, etc) to calculate the forces. The haptic feedback to the user's actions are calculated in real time and depend on the interaction between the virtual devices and the virtual anatomy (vessel geometry and vessel properties) and give realistic tactile feedback to the operator during the training procedure.16 ,17

Figures 2

The Vascular Interventional Training System (or VIST) full physics virtual reality simulation of the cardiovascular anatomy (A); the VIST simulator set-up in the skills laboratory (B) (courtesy of Mentice AB, Gothenburg, Sweden) and the fluoroscopic image of the aortic arch (with catheter handling error in situ) (C) and the VR simulated aortic arch (with catheter handling error in situ) (D).

Proficiency definition

Proficiency levels were established on experts’ mean performance on a specific CA VR case (case 2) on the VIST simulator, V.6.0. This methodology was first reported by Seymour et al5 and is described in detail elsewhere.18–20 Intraoperative defined errors included tool vessel errors such as severe dragging of the tip of the catheter along the wall of a vessel for a distance >3 mm. A catheter movement error was defined and recorded when the catheter was advanced into the carotid artery without the guide-wire tip inside the catheter or if the catheter was too close to the lesion. These metrics had been previously validated using a large number of trainees learning carotid angiography.8 ,17 The mean score used to define proficiency levels for the experts was fluoroscopy time ≤6.2 min and total technical error score ≤4.


The study was approved by the Institutional Review Board (Emory University, Georgia, USA). Simulation subjects were trained to reach a metric-based level of proficiency before completing their first procedure. Proficiency levels were defined on the objectively assessed performance of five US-based intravascular faculty experts (ie, had performed >1000 carotid angiograms) using the VIST case 2, V.6.0 and with the same metrics.

On the basis of prospective randomisation group A were trained on the VIST until they reached the quantitatively defined level of proficiency (figure 1). The other half (group B) completed a supervised and mentored training during an elective in vivo case according to the traditional mentor-apprenticeship learning model. Group B standard trained subjects were mentored for one complete CA case by a cardiologist very experienced in CA and stenting procedures. Both groups then completed a separate supervised but unmentored complete CA case functioning as the primary operator, proctored by an experienced interventional cardiologist who was also an expert on CA, but who was blinded as to the training status of the subject. The proctor was instructed to behave the same towards the subject as they would during a normal case.

The operative performance was video-recorded for subsequent analysis by experienced operators described above, who were blinded as to the operator's identity and training status. Video assessment was scored and analysed for unambiguously defined metric errors (box 1), attending takeovers, procedure time and fluoroscopy time for both groups. The mean inter-rater reliability (IRR) for metric assessment was 0.98 (range=0.88–1.0).

Box 1

Metric descriptions for carotid artery angiography intraoperative assessment

  1. Times: Total Procedure time.

  2. Fluoroscopy time.

  3. Total Volume of contrast used (not used in this study).

  4. Appropriate sequence of procedure—diagnostic shots—aortic arch, selective angiography of the common carotid artery (CCA), intracerebral angiography.

  5. Number of diagnostic catheters used to obtain diagnostic pictures.

  6. Assessment that the diagnostic catheters are coaxial with the lumen of the vessel in which angiography is being performed.

  7. Number of strategies used to deliver either the guide or sheath to the CCA.

  8. The sheath/guiding catheter may not be moved or migrate from the position at any time during the procedure.

  9. Catheter vessel errors, tip of the catheter scraping against the vessel wall while it is being advanced.

  10. Catheter positioned too close to the lesion.

  11. Catheter advancing uncontrollably into the lesion.

  12. Catheter advancing without a guide-wire in front of it.

  13. Quality of diagnostic images—need to assess whether the view that produces the least foreshortening of the lesion has been obtained.

Statistical power calculations

Power calculations were based on a previous VR to OR study using an almost identical experimental design.5 Previous studies found that the mean number of errors enacted by the VR trained group was 1.9 (SD=0.5) and by the Standard trained group was 7.38 (SD=2) with eight subjects in each group, that is, a 74.3% difference. Thus the statistical power of a 40% difference between the groups was calculated for N=6 in each group (ie, VR trained group hypothesised mean=4.428 (SD=2.0) vs Standard trained group mean=7.38 (SD=2), an α of 5%, a β of 50%) was found to be 95.1% for a two-tailed test.

This study was an investigation of a training technique, and thus there was no deviation in patient care from the normal standard of care in a teaching hospital. The patients were selected for CA. Anatomic exclusions criteria from the training process were patients who exhibited complex anatomic subsets deemed too difficult by the trainer for the primary operator.


The mean amount of time to perform the carotid artery angiography (CA), fluoroscopy time and operator intraoperative errors are shown in figure 3. Differences between the groups were compared with one-factor analysis of variance (ANOVA). Cardiologists trained to proficiency on the simulator (group A) performed the procedure faster (CI 18.13 to 35.67 vs 24.47 to 40.21) and used less fluoroscopy time (CI 10.82 to 16.48 (CI vs 12.58 to 22.13) than colleagues who were not trained on the simulator (group B) but not statistically significant better (procedure time F (df=1, 10)=1.41, p=0.262; fluoroscopy time F (df=1, 10)=2.948, p=0.117). However, group A simulator trained operators made significantly fewer objectively assessed intraoperative errors than their traditional training group B colleagues (F (df=1, 10)=20.791, p<0.001; CI 5.02 to 10.31 vs 11.87 to 18.46). Overall, the VR trained cardiologists performed the procedure 17% faster, used 21% less fluoroscopy and made 49% fewer intraoperative errors than standard trained colleagues (ie, ToT=17–49% difference between the two groups).

Figure 3

The mean and SE time that it took to perform the carotid angiography, amount of fluoroscopy used and number of objectively assessed intra-operative errors made by the standard trained cardiologists (n=6) and the VR trained interventional cardiologists (n=6).


This is the first study to demonstrate in a prospective, blinded and randomised way that experienced physician operators can be trained in a new procedure on a VR simulator. It also demonstrates that the VR training improves the operator's procedural skills and transfers those improved skills to performance of that new procedure on actual patients. Experienced interventional operators trained to a quantitatively defined level of proficiency on the simulator took less time to perform the procedure and used less fluoroscopy than their conventionally trained colleagues. Most importantly, the VR-trained experienced physician subjects also made significantly fewer objectively assessed intraoperative errors when performing their first case, solo, on actual patients. These results indicate that in this study there was (on average) 17%, 21% and 49%, respectively, transfer of training from the VR simulation training to the in vivo procedures. Furthermore, the results from this study show greater ToT than a similar study on experienced laparoscopic surgeons.21 One possible explanation is that in the previous study a lower fidelity VR simulation training device was used than in the study reported here. Furthermore, in both studies, the largest impact of VR training on procedural performance was on the significant reduction on objectively assessed procedure errors.

This study has implications for all procedural medicine and could open the door to a new standard for practising physicians attempting to learn new complex interventional techniques. A simple fact of modern healthcare is the evolution of newer and more complex, less invasive procedures for the treatment of disease. However, practising physicians need to continually update and learn new skills in order to master the new procedures. Additionally, population growth and an ever ageing population will only escalate this skill-procedure expansion further. Timely and convenient access to new procedural treatments by well-trained physicians will further challenge an already stretched training system which has started to realise that skill acquisition for the early part of the learning curve on patients is no longer acceptable.1 ,22 A significant part of the new procedural training challenge will be helping experienced physicians acquire the appropriate skills to perform new procedures or learn to use new devices safely without putting patients at risk. More than a century after Thorndike and Woodworth23 proposed that what was learnt in one context would transfer to a different but similar situation, the so-called ToT, we now have a training mechanism that can accomplish this goal. This concept underpins the efforts and philosophy behind the development of VR simulation methods for training in so many other contexts, for example, aviation, military,24 which now can be applied to new procedural training in interventional medicine.

However, on a cautionary note, not all VR simulations are equal. Simulation training must be more than just simulated experience supplanting repeated in vivo practice.19 Quality VR simulation training affords the trainee with the opportunity to engage in deliberate practice while making mistakes and giving immediate ‘proximate’ feedback when the mistake was made. This means that the trainees are given proximate formative as well as summative performance feedback which enhances and speeds the learning process. Gallagher20 has proposed that simulation should be defined as (1) an artificially created or configured ‘learning’ situation that allows for the practice or rehearsal of all or salient aspects of a procedure. Crucially, the artificial learning situation should (2) provide the span of appropriate sensory responses to learner physical actions that are behaviourally consistent with what would be experienced in real life (including the opportunity to enact both appropriate and inappropriate learner actions (ie, errors)). The simulation should also afford the opportunity to (3) perform the procedure (4) in the same order and (5) with the same devices with which the procedure would normally be performed. Thus, the fidelity of the VR simulation and correct simulation curriculum are critical for appropriate skills transfer to the trainee. That said, simulation-based training will never completely replace the in vivo clinical training experience. Rather, the function of simulation-based training (with the highest fidelity that is reasonably achievable) is to supplant the early part of the learning curve.

Nallamothu et al25 reported operator outcomes in 24 701 procedures by 2339 operators for carotid artery stenting with embolic protection and showed a clear operator annual volume/outcomes relationship. Evidence now shows that procedural skills can be maintained with intermittent VR simulation training.26 VR simulation models for skills training work because they provide a context, organisational structure and focus so as to allow for information to be easily retrieved from long-term memory.27 Interventionists can also practise/rehearse the sequencing of psychomotor skills to complete the task effectively, efficiently and safely with a highly structured, proximate, formative performance feedback, thus reducing the rate of skills loss that would naturally occur with non-use.28

The implications of the study reported here are considerable for new procedural skill acquisition as well as for maintenance of skills and competency assessment in procedure-based medicine disciplines.28 There is a growing body of evidence suggesting that the relationship between outcomes and procedure volume may not be as straightforward as previously thought29 and may in fact be better correlated to skill levels as well as other factors related to hospital quality controls and the support systems available for the delivery of care. One of the limitations of this study is that we relied on the volume of procedure experience as a surrogate for skills. The clinicians who participated were, however, so experienced with angiographic and interventional procedures that excellent ‘procedure’ skills were a reasonable inference. In addition, our study suggests that the additional use of high fidelity VR simulation can afford very experienced practising physician operators with the opportunity to acquire and maintain new procedural skills.

One of the limitations of this study was the small subject numbers, though even with small subject numbers the results demonstrated a large effect size which replicates the findings from previous studies but in different contexts.5 ,21 ,30 ,31 Additionally, although there was widespread enthusiasm expressed for the study by many colleagues, they were less forthcoming about enrolling and engaging in a randomised training study which objectively measured operator performance. This fact has implications and insight into an important issue, explaining why there might be resistance by very experienced physicians in learning new procedural skills in a more formal, quality-assured setting where actual operator skill and measured performance are assessed.


This report is the first study in medicine to demonstrate in a prospective, blinded and randomised way that experienced physician operators can be trained in a new procedure on a VR simulator and that the VR training improves the operator's procedural skills and transfers those improved skills to performance of that new procedure on actual patients. The results from this study have implications for new procedural skill acquisition. They suggest that full physics VR simulation may be a more effective way of very experienced physicians acquiring the skills for performance of a procedure novel to them. To date, simulation training has primarily been aimed at skill acquisition for less experienced operators. The data from this study suggest that this technology may have wider applications.


View Abstract


  • Contributors CUC and AGG were responsible for study design and data collection. CUC, LL and AGG were responsible for data analysis, writing the manuscript, results interpretation, paper critical revisions and agreed on final draft.

  • Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests LL has served as a consultant for Mentice AB, the manufacturer of the simulator used in the study. AGG is a member of the Editorial Board of BMJ Simulation and Technology Enhanced Learning.

  • Ethics approval Emory University Institutional Review Board.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement The data reported in the article are available by emailing AGG.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.