Major Incident Hospital Simulations in Hospital Based Health Care: A Scoping Review

Abstract

Major incidents are occurring in increasing frequency, and place significant stress on existing health-care systems. Simulation is often used to evaluate and improve the capacity of health systems to respond to these incidents, although this is difficult to evaluate. A scoping review was performed, searching 2 databases (PubMed, CINAHL) following PRISMA guidelines. The eligibility criteria included studies addressing whole hospital simulation, published in English after 2000, and interventional or observational research. Exclusion criteria included studies limited to single departments or prehospital conditions, pure computer modelling and dissimilar health systems to Australia. After exclusions, 11 relevant studies were included. These studies assessed various types of simulation, from tabletop exercises to multihospital events, with various outcome measures. The studies were highly heterogenous and assessed as representing variable levels of evidence. In general, all articles had positive conclusions with respect to the use of major incidence simulations. Several benefits were identified, and areas of improvement for the future were highlighted. Benefits included improved understanding of existing Major Incident Response Plans and familiarity with the necessary paradigm shifts of resource management in such events. However, overall this scoping review was unable to make definitive conclusions due to a low level of evidence and lack of validated evaluation.

Terrorism events, floods, bushfires, and even pandemics are occurring in increasing frequency over the past century. ^1,2 In health care, these events can be grouped under the term major incident (MI). An MI can be defined as “an incident or event where the location, number, severity or type of live casualties, requires extraordinary resources” beyond the normal “resources of the emergency and health care services’ ability to manage.” ^2–4

Over the past decades, preparation for MIs has become a focus of concern for health-care systems. During such events, hospitals must “adapt to exceptional situations, and all activities must be coordinated to cope with the unavoidable chaos… Everything is different from routine, and responders need to be coordinated by people accustomed to these dynamics.” ¹

These events must be analyzed from a systems perspective, to appreciate the complexity involved. A system can be defined as “a group of interacting, interrelated and interdependent components that form a complex and unified whole.” ⁵ Systems thinking provides a set of tools to describe and analyze these networks. It is particularly useful in addressing complex problems, that cannot be solved by any 1 stakeholder. It focuses on organization learning and adaptive management, and is a vital tool in addressing complex public health issues, such as MIs. ⁵

To reduce the chaos of these complex events, most Western health-care systems have developed a Major Incidence Response Plan (MIRP). ^3,6 Generally MIRPs are rarely “stress tested” and often not known by most staff. ^7,8 Practically, and ethically, it is only possible to test MIRPs by means of simulation. Thus, the methods to create high level scientific evidence are very limited. ⁹ In an MI simulation, the participating system “simulates the influx of a large number of patients” and the system responds to this stress. ¹⁰ Simulations vary in fidelity and scale. ^11,12 Ideally simulations should be evaluated, and learnings fed back into the involved system in a Plan-Do-Study-Act cycle. ^10,12

Anecdotally, MI simulations are thought to help improve health-care system preparedness, ^13,14 although it is difficult to objectively evaluate. ¹⁵ The majority of MI simulation research focuses on Emergency Department (ED) Triage, or prehospital care. ^16–21 However, analyzing MI response from the perspective of a single department does not reflect the impact of these events on the hospital system as a whole. For example, after the 2005 London Bombings, the Royal London Hospital stood down from the formal declaration of an MI 5 h after the bombings started and reopened for normal services. However, at the time of reopening “theatres were operating to full capacity and the intensive care unit had not received the patients it had already accepted from the MI.” ²² Published expert opinion after this event identified that:

“Such actions have the potential to further overload pressured systems. Thus, the ongoing care of the patients admitted from the incident should form part of a major incident plan as the impact of their admission and treatment is beyond a period of a few hours.” ²²

Thus, to determine how an MI may impact the hospital health-care system, wider whole hospital simulations must be performed. Locally, there is little published Australian data on hospital disaster preparedness. ²³ Therefore, the aim of this scoping review of the international literature was to determine if whole hospital-based simulation improves hospital response capability to prepare for and manage major incidences, from an Australian health-care system perspective. A systems perspective was used in the analysis.

Methods

Search Strategy

A systematic style scoping review was undertaken in August 2022, according to the recommendations of the Preferred Reporting Items for Systemic Reviews and Meta-Analyses (PRISMA) and the Joanna Briggs Institute (JBI) Methodology, with the aim to tabulate all relevant literature. ^24,25 The initial research question was reviewed against the population/problem, concept, and context (PCC) and FINER frameworks. ^25,26 This research aimed to determine if whole hospital-based simulation improves hospital response capability to prepare for and manage major incidences, from an Australian perspective. A systematic search was then undertaken, using 2 databases: PubMed and CINAHL. An attempt was made to include the ERIC database; however, no results were returned.

As per the JBI methodology, an initial limited search was undertaken in each database to identify appropriate key and index terms. A second formal search was then performed, and these results were included. Slightly different search terms were used between databases, due to different tools offered by each. The ERIC database was included in the initial limited search; however, no appropriate results were returned despite numerous searches.

The search terms are provided in Table 1, and the eligibility criteria can be observed in Table 2.

Table 1. Search terms

Note: Refer to Appendix 1 for full Boolean search string.

Table 2. Inclusion and exclusion criteria

The inclusion and exclusion criteria were predefined before beginning the scoping review (Table 2). Articles included must have evaluated the implications of the simulation on the health-care system (ie, not a pure feasibility study). Evaluation data must have been included. A broad scope of publication dates was included as these events are rare, and contemporary data were assumed to be minimal. Articles limited to the single departments were excluded.

The initial PubMed search returned 171 articles, and CINHAL returned 122 articles. These articles were combined then the title and abstracts were screened against the eligibility criteria. Refer to the PRISMA Flow diagram (Figure 1). After title and abstract screening, 54 relevant articles were identified for full text screening. Following this, 15 duplicates were identified and removed. Thus 39 articles proceeded to full text screening.

Figure 1. PRISMA flow diagram for systematic reviews. ²⁷

Prehospital or emergency department only simulation accounted for a significant proportion of articles returned in the search. However, these were not sufficient to answer the research question, and were excluded. Simulations based purely on mathematical and computational modeling were also excluded. This is justified by a 2008 study, which demonstrated that there were marked differences in patient benchmarks between computer simulation and live exercises. ²⁸ Three papers were excluded as they were set in Saudi Arabia, which was assessed as too dissimilar to the Australian population and health-care system. ^29–31 A further 3 studies were excluded as English translations were not available. ^32–34 Thus, after full text screening, 11 relevant articles were retained. Reference lists from the included articles were snowballed to identify relevant papers. However, no new articles were identified.

Quality Assessment

All included articles were assessed for quality against the appropriate CASP checklist. ³⁵ Of note, most articles were found to be of low evidence strength, likely due to ethical and procedural difficulties in this topic.

However, 2 studies were excluded for further quality concerns. A 2014 United States article was excluded, due to a very significant risk of selection bias and low strength of evidence. Self-reported perception of knowledge improvement was assessed by a post course questionnaire only, of which only 20 participants completed, despite a whole hospital simulation being conducted at 3 Los Angeles Hospitals with staff from all 3 hospitals participating. ³⁶ Another 2018 Dutch study was excluded as the primary outcome recorded was not considered valid by the authors of this review, and there were significant sources of bias. The original Dutch authors retrospectively evaluated 32 MI simulation reports from Dutch hospitals and identified the difference in the number of items of improvements identified in different reports. Measuring the number of areas of improvement identified, with no actual evaluation into these areas, is not a valid outcome measurement. The study was thus excluded as it lacks internal validity. ¹⁰ Please refer to Appendix 2 for further details.

Data Extraction and Synthesis

Author 1 of this review independently reviewed the relevant articles identified from the search strategy, described above. As per the JBI protocol for scoping reviews, ²⁵ data were extracted from each article under key characteristics and main conceptual categories.

Results

Study Characteristics

After a scoping systematic literature search, and the application of the exclusion and inclusion criteria listed in Table 1, a total of 11 relevant articles were identified, as can be seen in Table 3. Although the date range for inclusion was set as the past 20 y, the majority of articles (n = 10; 91%) were published in the past 12 y. Only 1 article included was based in Australia. Of the other articles, 4 were based in Sweden, 3 in the United States of America (USA), 2 in Italy, and 1 in England. The type and size of simulation used in the articles varied greatly, from tabletop exercises to multijurisdictional simulations. Where described, all simulations appeared to have involved a man-made MI.

Table 3. Characteristics of 11 included studies from international scoping review

Of the included articles, 8 were prospective observational study designs, 2 used quasi-experimental study design, with pre- and postsimulation evaluation. All 11 articles examined mixed populations, including both adult and pediatric patients. Assessed against the National Health and Medical Research Council (NHMRC) Evidence Hierarchy, 10 articles were all found to be of level 4 evidence with a high chance of bias. ³⁷ One study, a prospective cohort design which examined a purely pediatric cohort, was found to be level 3 evidence. ² Overall, there was a significant paucity in high level data.

Across the 11 included articles, there was a significant amount of heterogeneity in study designs, outcome measures, and evaluation techniques. No 2 articles used the same evaluation technique or outcome measures, making direct comparison difficult. In addition, a mixture of qualitative and quantitative measures were used across articles.

Common Themes Identified

The aim of this scoping review was to determine if whole hospital-based simulation improved hospital response capability to prepare for and manage MIs, from an Australian health-care system perspective. From a single site outlook, the 2020 Italian article provides the best example. ¹ Over a 2-y period, 7 whole hospital simulations were held using a preestablished course to train staff on the implementation of the hospitals MI plan. Overall, the authors found it to be an efficient way to train hospital staff in MI management, although the article was assessed as representing a low level of evidence.

This is supported by the other articles. In general, participants in the simulations self-reported improvement or increased understanding. ^1,7,38 Of interest, in a 2022 English article which involved a whole hospital simulation with more than 700 staff participants, further exercises were requested by the participants. They found “the simulations mimicked real responses and that exercising as a whole system was beneficial.” ⁷ The only Australian study located in the literature that used a whole hospital simulation found that participation in MI simulations improved factual knowledge among participants. ³⁸ Benefits of MI simulation reported in the included articles have been summarized in Box 1.

Box 1. Benefits of MI simulation

• - Improved understanding of roles in an MI ^1,7,39

• - Improved understanding of MIRP ^1,7

• - Familiarity with paradigm shift of managing resources to maximize survival ^1,7,39,40

• - Identification of latent errors and systems safety issues ^7,11,40,41

• - Identification of areas of improvement ^7,11,40–42

• - Testing surge capacity from a resource perspective ^11,40–42

• - Testing clinical tools for MI ²

Some articles evaluated an entire region’s response to MI by means of simulation. For example, the 2012 USA prospective observational study completed a full-scale regional exercise, which included 17 participating hospitals. All 17 hospitals considered the simulation exercise outcomes across the whole hospital. This massive exercise was used to evaluate the region’s response and identified key areas that required improvement. Similar areas of improvement were identified in the other 11 included articles; these have been summarized in Box 2.

Box 2. Areas of improvement identified by MI simulation

• - Communication ^7,11,40,41

• - Lack of working knowledge of MIRP ^7,11,39–41

• - Staffing, and medical resources ^7,11,40,41

• - Command structure ¹¹

• - Lack of compatibility between prehospital and hospital teams, or between departments. ⁴¹

• - Improved security during events ³⁹

• - Engagement with community partners and first responders ^39,43

• - Documentation ⁷

• - Media strategy ⁹

Some articles identified unique points, through more novel study designs. Refer to Appendix 3 for further information.

Discussion

Improving MI preparedness and management is a topic of significant public health concern. However, there is little published data evaluating management in real-world events. Some recommendations have been published after specific events; but these are examples of expert opinion only. ^{13,14,45–47}

Simulation has long been thought to be an effective tool to assist this preparation, although it is difficult to objectively evaluate. ^13–15 Unfortunately, similar to previous publications, ⁴⁸ this scoping review has also demonstrated a paucity of strong data. Studies were generally either quasi-experimental or prospective observational design. Although they contribute preliminary insights, these designs do not have randomization, a limited control of confounding variables, and no control group. This weakens the scientific strength of the evidence, and it must all be interpreted with caution.

In general, retrospective self-evaluation demonstrated improvement of MI simulation management, and increased understanding of MIRP. ^1,7,38 Participants in a 2022 study stated that “the simulations mimicked real responses and that exercising as whole system was beneficial.” ⁷ Thus simulations seem to improve staff confidence, which is important and beneficial. While performance is not a substitute for capacity, “individual, leader, and team confidence play essential roles in achieving success and the absence of confidence has been connected with failure.” ⁴⁹ Simulations appeared to be useful tools for identifying areas of improvement, as can be seen in Box 2. While these studies were highly heterogenous, similar themes of improvement were found, suggesting potential generalizability.

Simulations of a variety of fidelity were performed. Due to common deficiencies across the region, the 2012 USA study found that “tabletop exercises are inadequate to expose operational and logistic gaps in disaster response. Full scale regional exercises should routinely be performed to adequately prepare for catastrophic events.” ¹¹ From a systems perspective, it would be ideal to regularly run large scale exercises to truly stress the networks involved. However practically these exercises are expensive, time and resource consuming. ¹² Other studies used lower fidelity techniques as they believed “the resource investment and expense of high-fidelity simulation was not justified.” ⁷ At this stage, there is not enough evidence to support 1 approach over the other. However, despite fidelity level, all studies included found some benefit or identified areas of improvement.

As identified in the 2010 Swedish study “monitoring health-care quality may be difficult without the use of clinical indicators.” ⁹ This is further emphasized by the existing literature on MIs and simulation, which has found demonstrating the effectiveness of such exercises difficult. ^10,12 In this literature review, all studies evaluated their simulation differently. In future, to accurately evaluate the effectiveness of these activities clinical indicators must be developed. The proposed indicators in the 2010 Swedish study are 1 possibility, but they must be externally validated.

Review Strengths and Limitations

This is the first known scoping review on MI simulations in hospital-based health care that considers a whole hospital or regional response to MIs. It provides preliminary insights into the areas of benefit and possible improvements that could be made to MI simulation. To ensure rigor in our process, this scoping review followed the JBI manual, carried out pilot searches to refine search terms, and predefined inclusion and exclusion criteria before screening.

However, the generalizability of these scoping review findings to different international health-care systems is a limitation of concern. Only 1 study identified in this review was performed in Australia. Four studies were performed in Sweden, and 1 in the United Kingdom. Arguably, these countries have comparable health-care system. ⁵⁰ However, this review also included 3 American studies, which has a vastly different health-care system and limits the generalizability of the American study findings. ⁵⁰ Thus, conclusions from these articles must also be interpreted with caution, when considering within the context of different health-care systems. This concern is reinforced further by acknowledging the essential role and influence of the key elements of the systems thinking framework.

There were other limitations to this scoping review. The database search was performed by a single author, which may introduce a bias regarding the “relevant” articles included. Additionally, the author was unable to include or analyze 3 articles published in another language. ^32–34 Another limitation that should be acknowledged is the small number of included articles; however, this may be reflective of the current literature deficit in this field.

To support the value of simulation in MI preparation and management, further research must be performed. Specifically clinical indicators of MI management should be validated, which would allow more scientific and objective evaluation of MI simulation in the future.

Conclusions

This scoping review of the international literature aimed to determine if whole hospital-based simulation improves hospital response capability around MIs. Definitive conclusions were unable to be made, due to the low number of relevant articles identified, the lack of data, and the general paucity of strong scientific evidence. In general, all articles had positive conclusions with respect to the use of MI simulations. Several benefits were identified, and areas of improvement for future highlighted. However overall, there was a lack of validated evaluation, little evidence to definitively conclude that simulations improved preparation or management for real world MIs. Further research is required to optimize future responses to MI events.