Article Text

Download PDFPDF

A qualitative evaluation of the role of simulation in policy development for service improvement
  1. Thomas Blanks1,
  2. Nicholas Woodier1,
  3. Bryn Baxendale2,
  4. Mark Fores1,
  5. Lynn Fullerton3
  1. 1 Trent Simulation and Clinical Skills Centre, Medical Centre, Nottingham, UK
  2. 2 Nottingham University Hospitals NHS Trust, Trent Simulation & Clinical Skills Centre, QMC Campus, Nottingham, UK
  3. 3 Practice Development Matron, Nottingham City Hospital, Nursing Development, Derwent House Nottingham, Nottingham, UK
  1. Correspondence to Dr Thomas Blanks, Trent Simulation Clinical Skills Centre, Queen’s Medical Centre, Nottingham, NG7 2UH, UK; thomas.blanks{at}nhs.net, drtomblanks{at}gmail.com

Abstract

Objective To evaluate the efficacy of simulation-based techniques to prospectively assess developing polices prior to implementation.

Methods A self-selected sample of nursing staff from a local, acute hospital reviewed a draft intravenous drug administration policy before simulating drug administration of either an infusion or direct injection. The participants completed a postsimulation questionnaire regarding the new policy and simulation, took part in a semistructured interview and were observed during the simulation with their consent.

Results 10 staff attended the simulation. The emergent themes identified a wide range of factors relating to the everyday usability and practicalities of the policy. There were issues surrounding inconsistent language between different clinical teams and training requirements for the new policy.

Conclusion Simulation, using simple scenarios, allows the safe evaluation of new policies before publication to ensure they are appropriate for front-line use. It engages staff in user-centred design in their own healthcare system.

  • healthcare ergonomics
  • simulation
  • policy
  • user testing
  • Implementation

Statistics from Altmetric.com

Introduction

Policy development, publication and dissemination are challenges that are faced regularly throughout the National Health Service (NHS) and many other healthcare models. The application of policies reflecting latest research can often be difficult to apply to front-line staff in healthcare.1 2

There is limited literature on how to improve general policy compliance; however, much work has been done on specific policies such as hand washing to investigate barriers to adherence.3 4 The general aspects of these include high workload, understaffing, perceived lack of time, lack of a role model and lack of active participation in promotion at an individual level. Unrealistic workplace goals have been highlighted as a contributory barrier to nursing staff participation in development projects.5

With these barriers to adherence, policies that are deemed unworkable by staff are promptly violated with both potential legal and financial implications both to the individual and the system. The Health and Safety Executive report ‘Improving Compliance with Safety Procedures Reducing Industrial Violations’6 explicitly mentions that impractical rules encourage violations.

We have taken an innovative approach to policy development in light of a series of medication-related incidents at our local NHS Trust. By making effective use of simulation combined with qualitative analysis of audio and visual recordings of nursing behaviour when trialling a new policy. we can gain prospective and dynamic feedback on an updated policy prior to mass implementation. This builds on previous publications relating to the use of simulation to predict impact that a change may have on a healthcare system.7 This has been shown to be effective in other areas of behavioural change and assessment in a practical learning environment particularly concerning knowledge and understanding of complex healthcare reforms.8 Our project is very much based on this work—the basis being exposure of participants to a realistic environment and then assessing their response to a situation. The minimisation of variables key to simulation practice is uniquely suited to this type of policy assessment. This is turn will avoid the all-too-common occurrence where an implemented policy is widely ignored by the front-line staff due to pragmatic considerations. These can include time, personnel, qualification of staff, system pressure or workload.

The role of simulation-based training is well established for the learning of both technical9 and non-technical skills.10

This initial phase will precede a formal policy development role for the Trent Simulation and Clinical Skills Centre (TSCSC). The focus of this study was the local Trust’s draft1 intravenous drug administration policy. This policy was under review by the Trust with exploration of the potential to allow single-person checks for certain medications.

This manuscript was written with reference to the Revised Standards for Quality Improvement Reporting Excellence guidelines11 where applicable.

Objective

Evaluate the efficacy of simulation-based techniques to prospectively assess developing polices prior to implementation.

Methods

Recruitment

Recruitment strategy

Table 1

Clarification of staff attending pilot session

  • This study recruited from the local NHS Trust’s Divisional Nursing staff.

  • This group includes multiple levels of seniority and all specialties of nursing staff, midwives and operating department practitioners.

  • Self-selected volunteers based from the above pool.

Sample size

  • The pilot phase was run with 10 volunteers from the above pool.

  • The breakdown for volunteers is shown in table 1. As can be seen, there was limited involvement from band 5 nursing staff, which will be addressed shortly.

Ethical approval

Use of the Medical Research Council NHS Health Research Authority ethical decision tool12 confirmed that ethical approval was not required for this study.

This was confirmed locally with the local Research and Innovation department.

Study design

This study used a simulation-based programme to evaluate the new draft intravenous drug administration policy. Data were collected at several points throughout the process.

Data were collected from candidates via non-participant observation and semistructured interviews. It was decided that a qualitative review would be better suited to explore the feedback and behaviours exhibited during the simulation. The nature of the simulated environment allows deep exploration of an individual participant’s non-technical skills and behaviours13; qualitative analysis is ideally suited for this.14

On the day of the simulation, nursing staff were invited individually and in pairs to attend TSCSC. The process for each participant was then

  1. Arrive, presimulation questionnaire (see online supplementary appendix 1) and briefing.

  2. Informed consent and signing of consent form for audio-visual recording.

  3. Fifteen minutes time to familiarise themselves with policy. Participants were given a copy of the full draft policy and also an outline of the policy highlighting steps in administering intravenous medications and when only a single-person check may be required.

  4. Undertaking of the simulation (video recorded). Participants were allocated to undertake one or both of the following scenarios:

    1. Administration of a maintenance infusion of aminophylline. This required a two-person check.

    2. Administration of a direct injection of co-amoxiclav. The draft policy was exploring whether this medication could be administered with only a single-person check.

  5. Immediate debriefing after the simulation, audio recorded, using a standard approach (see online supplementary appendix 2).

  6. Postsimulation questionnaire using five-point Likert scale methodology15 (see online supplementary appendix 1).

  7. Close and thanks, including reminding staff that they have been considering a draft policy here and they should still follow current Trust policy.

Supplementary Material

Supplementary data

A simulated patient was used for the simulations. Medications were injected into a manikin arm attached to a real human who acted as the patient.

Data collection

Precourse questionnaire

This is fully outlined in online supplementary appendix 1. The questionnaire focused on the participant’s perceived awareness, understanding and effective use of the current intravenous Trust policy. Both the questionnaire and semistructured interview questions were adapted from Nielsen’s heuristic assessment.16 Although the Nielsen assessment is principally concerned with user interface design, they provide a useful framework for exploring general usability in our context.

Simulation observation

Observation was conducted mainly by a single author (TB or NW), with a smaller proportion conducted by both (TB and NW) in order to maintain internal validity. Contemporaneous notes were taken during the observational piece.

Both audio and visual recordings were taken and used with the candidates’ consent. These in turn were then analysed by two independent reviewers (NW and TB) for the following:

  • adherence to new policy guidelines with specific focus on the changes to current guidelines (eg, suggestions that some medications might be a single-person check)

  • inappropriate deviations from the new policy.

Semistructured debrief interview

Fully outlined in online supplementary appendix 2, this interview explored the candidate’s thoughts and experiences of both the new policy and also the use of simulation for policy development.

The interviews were recorded with detailed contemporaneous notes which were transcribed at a later date. These notes were then closely read by the two authors (TB and NW) and recurrent themes were generated from this. The process of thematic review for qualitative data has been widely used within other scientific fields as a flexible and useful tool for describing a wide-ranging data set that qualitative data provides.17

Postcourse questionnaire

Analysed with the precourse questionnaire and emergent themes transcribed.

Results

Questionnaire feedback

The feedback from participants was positive regarding the ease of access and clarity of the new policy. However, there were significant concerns regarding the feasibility and practicality of the new policy. There was a consistent theme to the feedback regarding the importance of front-line staff involvement in policy development as well as the perceived usefulness of simulation in policy evaluation.

Analysis of the emergent themes of the free-text comments related to the policy showed significant anxiety towards the removal of a second checker, general discomfort with the principle of a single-person checking system. Other emergent themes are listed below:

  • a large document with potentially difficult access

  • lack of awareness when new policies are introduced or updated

  • variability of policy interpretation in different clinical areas.

Analysis of the free-text comments related to simulation aspect of policy evaluation was uniformly positive, as the summarisation would suggest. The effectiveness of simulation when combined with a semistructured interview afterwards for a debriefing component was particularly mentioned.

Non-participant observation and semistructured interview

The participants were observed in simulated practice (see online supplementary appendix). Any inappropriate deviations from the current policy and aseptic non-touch technique (ANTT) were highlighted back to participants by the nursing staff involved. Table 2 categorises the emergent themes and comments that were collated from faculty observation and the semistructured interviews and shown below.

Table 2

Collation of faculty observation and debrief findings organised into themes

Limitations

It is important to identify the limitations of the data collected in this evaluation. While the results described are valuable in developing the policy, they should in no way be considered conclusive and provide a snapshot of a limited population. Key limitations were

  • Small scale

  • This was a small-scale evaluation and the staff attending did not provide a representative sample of all staff groups and clinical areas that administer intravenous medications.

  • Staff experience

  • The staff that attended the simulation were mostly either experienced staff or student nurses. There was limited representation from band 5 nursing staff who would regularly be undertaking the task of intravenous administration.

  • The financial situation facing the NHS has been well documented.18 19 As such, the availability of staff for non-mandatory training has been limited with a subsequent impact on this study.

  • Simulation

  • The evaluation of this policy was through simulation. While this provides an effective mechanism for evaluating policies, there are differences between simulated and real-world environments.

Discussion

Despite the aforementioned limitations, even a short simulation event provided a wealth of information to support the development of the new intravenous skills policy. It also highlighted that staff wished to be more involved with policy and guideline development, and that simulation is an effective tool to support development. Users of policies and guidelines should always be included in the development process because, often, the way a policy is written does not reflect the potential influences on policy adherence on the front line.

This study has potential significant impact on how policy development and implementation occurs. As discussed previously, there are multiple and varied reasons for failure of policy implementation, awareness and compliance ranging from lack of education to impractical use requirements. This study allowed the investigators to identify some of these latent conditions towards failure of successful implementation. Those specific to this policy were the following.

Ease of access to and visibility of key information

Certain pertinent information was not easy to find even with paper copies of the policy to hand and time dedicated to reading said copies; this was reflected in the simulated session.

Clarity surrounding the process and changes to the process

Potential changes to the process were unclear and so staff defaulted back to what they were familiar with.

Consistent language

The language of the policy referred to ‘direct injection’ and ‘bolus’ interchangeably and with inconsistent variability; this highlighted a disconnect between the language used by pharmacy and nursing staff.

Practical application

Many staff felt the new changes were not feasible in practice

Training requirements

Staff highlighted that in order to train staff in the new policy a hands-on approach would be required rather than the usual ‘cascade’ approach to information dissemination.

Although it is possible that some of these may have been identified in the final amendment process, it is highly unlikely that the above issues would have been explored in such detail. This has given the policy implementation team an evidentiary base to further analysis and policy development; particularly focused on practical components and use. Ensuring that a high-profile change in policy is advertised to all staff and combined with hands-on practical learning opportunities with overcome the educational barriers.

Due to the feedback and participant evaluation, the specific policy evaluated has yet to be fully implemented due to widespread staff concerns regarding the lack of two-person checking. This in itself has demonstrated the importance of simulation to policy testing; as previously discussed, the implementation of an inappropriate policy does not reflect well on an organisation. The value of this process has been recognised at NUH for whom this has become a clear area of interest for further pursuit.

Further work

The semistructured interviews documented the variability in interpretation of the policy within different clinical areas. An exploration of the environmental climate in clinical areas that allow deviation from policy in order to evaluate what factors influence this would be of benefit. It is unlikely that any policy rewrite will succeed if a climate already exists that does not follow the current policy or allows deviation from it. A deeper understanding of why the policy is not followed is required.

Further work within the Trust has been focused on increasing the scale of this programme to accommodate further policy testing procedures. At this current time, there is a larger-scale project involving further drug administration policies being run, which will allow a broader range of policies to be evaluated for feasibility.

Conclusions

An appreciation of latent factors within healthcare design is becoming more mainstream within clinical spheres, particularly when identifying violations of policies as a factor in serious incidents.

Simulation, using simple scenarios, allows the safe evaluation of new policies and guidelines before publication to ensure they are appropriate for front-line use. It engages staff in user-centred design in their own healthcare system. It is the authors’ belief that all clinical policies that were suitable should be evaluated using simulation before publication.

Appendix

Supplementary material.pdf

References

Footnotes

  • Contributors All authors certify that they have participated sufficiently in the work to take public responsibility for the content, including participation in the concept, design, analysis, writing or revision of the manuscript. Conception and design of study: BB, NW, MF and LF. Acquisition of data: NW, LF and TB. Analysis of data: WN and TB. Drafting the manuscript: TB and NW. Revising the manuscript critically for important intellectual content: BB, LF and MF. Approval of manuscript to be published and accountability for manuscript: all authors.

  • Competing interests None declared.

  • Patient consent Obtained.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.