Development and Validation of a New Tool to Improve the Accuracy of the Hospital Mass-Casualty Incident Response Plan Activation: The PEMAAF Score

Abstract

Introduction:

Effective response to a mass-casualty incident (MCI) entails the activation of hospital MCI plans. Unfortunately, there are no tools available in the literature to support hospital responders in predicting the proper level of MCI plan activation. This manuscript describes the scientific-based approach used to develop, test, and validate the PEMAAF score (Proximity, Event, Multitude, Overcrowding, Temporary Ward Reduction Capacity, Time Shift Slot [Prossimità, Evento, Moltitudine, Affollamento, Accorpamento, Fascia Oraria], a tool able to predict the required level of hospital MCI plan activation and to facilitate a coordinated activation of a multi-hospital network.

Methods:

Three study phases were performed within the Metropolitan City of Milan, Italy: (1) retrospective analysis of past MCI after action reports (AARs); (2) PEMAAF score development; and (3) PEMAAF score validation. The validation phase entailed a multi-step process including two retrospective analyses of past MCIs using the score, a focus group discussion (FGD), and a prospective simulation-based study. Sensitivity and specificity of the score were analyzed using a regression model, Spearman’s Rho test, and receiver operating characteristic/ROC analysis curves.

Results:

Results of the retrospective analysis and FGD were used to refine the PEMAAF score, which included six items–Proximity, Event, Multitude, Emergency Department (ED) Overcrowding, Temporary Ward Reduction Capacity, and Time Shift Slot–allowing for the identification of three priority levels (score of 5-6: green alert; score of 7-9: yellow alert; and score of 10-12: red alert). When prospectively analyzed, the PEMAAF score determined most frequent hospital MCI plan activation (>10) during night and holiday shifts, with a score of 11 being associated with a higher sensitivity system and a score of 12 with higher specificity.

Conclusions:

The PEMAAF score allowed for a balanced and adequately distributed response in case of MCI, prompting hospital MCI plan activation according to real needs, taking into consideration the whole hospital response network.

Introduction

During a mass-casualty incident (MCI), emergency medical resources are rapidly overwhelmed by the number and severity of casualties. Effective response to such events entails the activation of MCI plans. ^1,2 A hospital MCI plan features different levels of activation, from a “standby level” to upper levels that require withholding routine activities to effectively manage the large influx of patients by re-organizing staff, stuff, and structure. ³ Since activating a hospital MCI plan might result in the cancellation of elective diagnostic and therapeutic procedures, as well as elective surgery and ambulatory activities, thus impacting the provision of basic health services for the population, medical responders and hospital managers should carefully weigh the decision of activating the plan. ⁴

In the current literature, many after action reports (AARs) of hospital responses documented the increasing need to avoid an excessive mobilization of hospital resources compared to the actual needs, preserving routine hospital activities, and avoiding unjustified expenses. ^4–8 The unpredictability and uncertainty that characterize MCIs certainly impact the decision-making process of first responders, who are often exposed to unconfirmed or contradictory information, especially in the first phase of the disaster response. ^4,6 This blurred scenario, together with the constant stress and time pressure, can lead responders to either rush into activating an inappropriate level of response ^9,10 or to wait too long before activating the MCI plan. ^11–13

Unfortunately, in the literature, there are no tools available to support hospital medical first responders in performing the initial assessment of the severity of an MCI, or to guide the activation of the hospital MCI plan, indicating the proper level of response.

In urban settings, where several hospitals with different levels of care are usually part of a dynamic response network coordinated by a local or regional emergency communication center or dispatch center (DC), coordination and communication among the different health facilities become crucial during an MCI. The lack of a tool able to guide a coordinated response within an urban hospital network further complicates the overall MCI management.

The Metropolitan City of Milan, the second most populous metropolitan city in Italy with its 133 municipalities and 3.25 million inhabitants (2019), has recently been affected by several MCIs. ^14,15 The AARs of those events reported that all the hospitals present in the area activated the higher function level of their MCI plan, as if each hospital was the only one involved in the response.

This manuscript aims to describe the scientific-based approach used to develop, test, and validate a new tool, named the PEMAAF score (Proximity, Event, Multitude, Overcrowding, Temporary Ward Reduction Capacity, Time Shift Slot [Prossimità, Evento, Moltitudine, Affollamento, Accorpamento, Fascia Oraria]), that could assist hospital managers and hospital medical first responders in the MCI management by predicting the required level of hospital MCI plan activation after an initial assessment, and could also assist crisis units or DC personnel by guiding a proper coordinated activation of an urban multi-hospital network.

Methods

Study Phases

To develop the PEMAAF score, the authors adapted the methodology used in the analysis of user requirements for information system design, which included the gathering of relevant information, the identification of user needs, envisioning, evaluating, and testing. ¹⁶ These different steps have been summarized and hereafter described into three distinct phases, encompassing a research phase for information gathering and identification of user needs, the conceptualizing and development of the score, and its validation through a series of evaluating and testing activities.

Study Setting

In the Metropolitan City of Milan, there are 35 hospitals serving a population of 3.25 million inhabitants. The majority of these hospitals are located in Milan, its capital city. In case of an MCI, the Metropolitan Emergency Medical Services (EMS) DC, (SOREU Metropolitana - Regional Emergency Agency Operations Rooms [Sale Operative Regionali - Agenzia Regionale Emergenza Urgenza]), notifies the hospital emergency departments (EDs) closer to the event. The regional law for major trauma network referral pathway classifies the 35 hospitals into four different levels: (1) specialized trauma centers (CTS); (2) local trauma centers with neurosurgery (CTZNCH); (3) local trauma centers without neurosurgery (CTZ); and (4) trauma emergency rooms (PST).

Within this referral pathway, MCI casualties are referred to the different health care facilities according to a series of pre-registered information collected in a confidential document deposited with the regional authorities, which include the number of staffed operating rooms available in the hospital and the number of triage codes that can be managed during the first three hours of the MCI.

Research Phase

The authors retrospectively analyzed the AARs documented by the Luigi Sacco University Hospital (Sacco Hospital; Milan, Italy) after the response to five MCIs, four fires and a railway accident, that occurred from 2016 through 2018. The Sacco Hospital, which is one of the 35 hospitals within the Metropolitan City of Milan, is a 500-bed trauma center classified as a CTZ according to the regional trauma network classification. According to its MCI plan (lastly revised in 2015 for the World Exposition held in Milan), the activation follows an “On-Off” mechanism after a direct notification from DC. The retrospective analysis of the AARs led to the identification of six key variables:

1. 1. Distance from the hospital (proximity indicator);

2. 2. Type of event;

3. 3. Number of people involved;

4. 4. Emergency department room (EDR) overcrowding rate;

5. 5. Temporary ward reduced capacity, either scheduled (eg, summer holidays) or unscheduled (eg, maintenance); and

6. 6. Time shift slot.

The authors all agreed that these six variables represented the main critical information that needed to be known at the hospital level to optimize the MCI management of the five events analyzed. ^17–19

This agreement was also supported by their own previous field experience and by the current scientific literature.

Score Development Phase

For each of the variables identified, the authors elaborated a series of possible options (eg, the proximity indicator the two options identified where either “inside the city of Milan” or “outside the city of Milan”). For each option, a specific value from one-to-three or from one-to-four was given to obtain the weighted decision matrix depicted in Figure 1. This matrix was named the PEMAAF score as the Italian acronym of the six variables identified and which constituted the matrix. As shown in Figure 1, the first version of PEMAAF score could range from five to 12 points. Three different levels were then identified and labeled by a color code (green, yellow, and red) representing the growing severity of the event and the complexity of the hospital response. A PEMAAF score below six corresponds to a “green alert,” which might suggest a reconnaissance of the situation in the ED. A PEMAAF score from seven through nine calls for a “yellow alert,” which might entail a series of procedures aiming to prepare the ED to manage the MCI casualties, such as early discharge of stable patients or patient transfers into hospital wards, as established by the hospital MCI plan. This second level of activation might allow hospitals to initiate preparatory activities without activating the whole MCI plan, thus safeguarding routine hospital activities. A PEMAAF score of ten or higher corresponds with a “red alert,” which might suggest the full activation of the hospital MCI plan, including the interruption of routine hospital activities, cancellation of elective procedure, and mobilization of additional resources.

Figure 1. PEMAAF Score (First Version).

Abbreviations: PEMAAF, Proximity, Event, Multitude, Overcrowding, Temporary Ward Reduction Capacity, Time Shift Slot [Prossimità, Evento, Moltitudine, Affollamento, Accorpamento, Fascia Oraria]; SOREU, Regional Emergency Agency Operations Rooms [Sale Operative Regionali - Agenzia Regionale Emergenza Urgenza].

Score Validation Phase

To validate the PEMAAF score and analyze its reliability, a multi-step process was followed:

1. 1. Retrospective analysis of past MCIs, using the score;

2. 2. Focus group discussion (FGD) to evaluate results and refine the score;

3. 3. Second retrospective analysis using the score modified by the FGD; and

4. 4. Final prospective study.

First Retrospective Analysis

In the first retrospective analysis, the authors used real data derived from the Sacco Hospital AARs to mimic the decision-making process and apply the PEMAAF score. Results were then compared to real decisions undertaken by responders during the five MCIs described by the Sacco Hospital AARs and presented to the DC in a dedicated FGD (Table 1).

Table 1. Retrospective Analysis of the Response to MCI at the Sacco Hospital 2016-2018: Real-Time Hospital Management Decision versus Response Suggested by the PEMAAF Score

Abbreviations: MCI, mass-casualty incident; PEMAAF, Proximity, Event, Multitude, Overcrowding, Temporary Ward Reduction Capacity, Time Shift Slot [Prossimità, Evento, Moltitudine, Affollamento, Accorpamento, Fascia Oraria].

Focus Group Discussion

The objective of the FGD was to review the application of the score, to identify gaps, and to establish corrective measures to ultimately refine the PEMAAF score. During the discussion, participants also reviewed data concerning nine MCIs declared on a regional basis, during the period 2016-2018, analyzing the type of event, the number and type of EMS vehicles involved (ambulance/car/helicopter), the location, time of the day, and the number of hospitals alerted. The PEMAAF score was also applied to the decision-making process of the nine additional MCIs and results were compared to real decisions of responders.

Second Retrospective Analysis

In the second retrospective analysis, the revised PEMAAF score (Figure 2) was applied to one of the five MCIs, the “Pioltello railway accident,” for validation. A dedicated panel of experts comprising the authors of this paper recruited to analyze results, decided to reproduce the abovementioned scenario in four different sites, situated at important railway junctions, respectively located in the north-east, north-west, south of Milan, and in the center of Milan. The 12 hospitals involved represent a reliable sample of all levels of trauma care alerted by the DC.

Figure 2. PEMAAF Score (Revised).

Abbreviations: PEMAAF, Proximity, Event, Multitude, Overcrowding, Temporary Ward Reduction Capacity, Time Shift Slot [Prossimità, Evento, Moltitudine, Affollamento, Accorpamento, Fascia Oraria]; CTS, specialized trauma center; CTZNCH, local trauma centers with neurosurgery; CTZ, local trauma centers without neurosurgery; PST, trauma emergency rooms; MCI, mass-casualty incident; SOREU, Regional Emergency Agency Operations Rooms [Sale Operative Regionali - Agenzia Regionale Emergenza Urgenza].

To validate the score, each of the six variables of the PEMAAF were considered as epidemiological units. The statistical analysis was based on simulated activations of the system. Firstly, two indexes were devised in order to validate the predictive power of the PEMAAF score during the process (Figure 3).

Figure 3. PEMAAF Score Indexes.

Abbreviation: PEMAAF, Proximity, Event, Multitude, Overcrowding, Temporary Ward Reduction Capacity, Time Shift Slot [Prossimità, Evento, Moltitudine, Affollamento, Accorpamento, Fascia Oraria].

These indexes can identify intervals linked to the saturation of the resources made available by the activated hospitals:

• > 1.0: over-saturation of the response system (especially if restricted to red and yellow codes);

• 0.75-1.0: saturation of the response system;

• 0.5-0.75: functionality adequacy of the response system;

• 0.25-0.5: excessive activation of the response system; and

• <0.25: improper activation of the response system.

In a following step, a minimum number of further replicas was established to create a statistically significant sample to validate the PEMAAF score, by using the G Power V3.0 Software program (Faul, Erdfelder, Lang, Buchner; Germany). Considering that the PEMAAF score consists of six variables (categorical, dichotomous, or polytomous with hierarchical order), for each simulation, 72 specific answers to the score’s questions were collected for each hospital. The following points were considered as critical: (1) the epidemiological unit (ie, the information per single variable); and (2) the bivariate correlation between what is suggested by the PEMAAF score and the effective activation of its hospital MCI plan. Assuming a probability of error α <0.05 and the probability of error β of 95%, the total of the minimum sample of observations collected was between 162 (slope correlation of 0.25) and 262 (slope correlation 0.20). ²⁰ The “curve estimation” method was used for the statistical analysis and identification of the scores, and, in particular, the regression with the Italic S correlation was identified as the “optimum.” The confirmatory test was based on the determination of Spearman’s Rho. The analysis was later extended to the 12 hospitals alerted for the MCI in Pioltello, Italy.

Prospective Study

In a subsequent step, the 12 hospitals were involved in four simulations to test the PEMAAF sensitivity and specificity, respectively held in 2019 on August 13, October 18, December 13, and on January 10, 2020. The four simulations were conducted via phone involving the DC and a member of the strategic management (strategic decision-making level), and the physician designated as coordinator of the hospital MCI plan (operational decision-making level), which were selected by each one of the hospitals involved in the activity. The first two simulations used data retrieved from the AAR of the MCI in Pioltello, a scenario well-known by the responders. In contrast, two additional scenarios had been designed by the authors for the simulation performed in December (an explosion in a department store facing Piazza Duomo in Milan) and in January (explosion of gas cylinders in the EXPO 2015 area), thus forcing responders to deal with an unknown scenario.

For the first two simulations, the hospital strategic management office provided to the DC only their PEMAAF reference on the second “A” item (temporary ward reduction capacity) by phone (affirmative or negative answer). The phone call was replicated at different time slots (11:00am, 6:00pm, 12:00pm, and 7:00am of the following Saturday morning) to calculate and test the PEMAAF score in the different time shift slots and the corresponding personnel. The last two simulations performed in December 2019 and January 2020 involved four different phases: (1) first alert call from the DC to the hospitals according to the hospital alert list and real-time filling out of the PEMAAF score by the MCI coordinator; (2) second call from the DC to the different alerted hospitals asking for confirmation of the level of alert chosen by the hospital health management (code suggested by the PEMAAF score or different code chosen according to hospital internal protocols and guidelines), communication of the hospital surge capacity by color code, and operating rooms and relevant from the MCI scene; (3) application of the SORT triage to the victims on the scene and distribution of patients according to the mapping obtained from the score (performed by the DC); and (4) third call from the DC to hospitals to communicate the number, color code, and injury mix of patients sent to the EDRs. During the simulated event, a one-way communication protocol (from DC to hospitals) was adopted, and patient influx to hospitals from the scene considered the EDR capacity declared in the confidential document by each hospital in case of MCI plan activation. In a last step, the PEMAAF score model had been verified and evaluated with respect to specificity and sensitivity through a regression model and using the statistical sensitivity test represented by Spearman’s Rho test or receiver operating characteristic/ROC analysis curves. The SPSS 20.1 (SPSS Statistic Software; IBM Corp.; Armonk, New York USA) version has been used for the statistical analysis with an established significance level of P <.05. The gap arising from the inverse relationship between specificity and sensitivity required a corrective tool, which led to the introduction of an additional level of alert, the “orange level,” associated with a PEMAAF score between 10 and 11. The purpose of this additional level was to allow hospitals to reach a very advanced stage of preparation with respect to the conservation of the discretionary activities in progress and guarantees better flexibility of the score in terms of resources to be activated (ready to cope) and capacity to cope with the influx of patients.

Results

First Retrospective Analysis

Table 1 compares the real decision undertaken by the Sacco Hospital during the five MCIs (Pioltello railways accident and four fires) against the action proposed by the application of the PEMAAF score. As shown, Sacco Hospital activated the MCI plan three times. Results indicated that the application of the PEMAAF score would have reduced the activation in two cases out of the three. Specifically, the activation of the hospital MCI plan was deemed as necessary only for the fire MCI that occurred at the Fire 4, while a “yellow alert” would have been enough in managing the fires in Fire 1 and Fire 2.

Focus Group Discussion

Figure 2 shows the final PEMAAF score obtained after revision performed by the expert panel. During the FGD, three points of re-evaluation emerged, concerning: (1) the “proximity” item, or the determination of two different kilometric cut-offs, considering the size of the Milan area; (2) the “event” item, aligning it to the level of care capacity/specialist resources for the treatment of trauma available in each hospital codified by levels of competence considering the complexity of traumatic injuries; and (3) the “ED overcrowding” item, integrating it with the software used by EDRs on a routinary basis.

Second Retrospective Analysis

The application of this updated version of the PEMAAF score to the Pioltello MCI confirmed its balancing effect on the hospital MCI plan activation and on the hospital general alert level, when compared with the decisions taken by the hospital management without the aid of the score, as shown in Table 2.

Table 2. “Pioltello” MCI Retrospective Analysis: Real Hospital Response versus PEMAAF Score Application

Abbreviations: MCI, mass-casualty incident; PEMAAF, Proximity, Event, Multitude, Overcrowding, Temporary Ward Reduction Capacity, Time Shift Slot [Prossimità, Evento, Moltitudine, Affollamento, Accorpamento, Fascia Oraria]; CTS, specialized trauma center; CTZNCH, local trauma centers with neurosurgery; CTZ, local trauma centers without neurosurgery; PST, trauma emergency rooms.

^a Hospital level according to organization of an integrated system for the trauma treatment (in Italian indicated with the acronym S.I.A.T Trauma Network).

Prospective Study

Four simulations and consequent assessment were performed from August 2019 through January 2020. For each simulation, 72 answers were collected to the score’s questions for each single hospital, thus obtaining a total number of 288 observations. As shown in Figure 4, the PEMAAF score was affected by time, suggesting that the hospital MCI plan was more frequently activated during night shift and the holiday shift.

Figure 4. PEMAAF Score Daily Trend According to Time Shift.

Abbreviations: PEMAAF, Proximity, Event, Multitude, Overcrowding, Temporary Ward Reduction Capacity, Time Shift Slot [Prossimità, Evento, Moltitudine, Affollamento, Accorpamento, Fascia Oraria]; CTS, specialized trauma center; CTZNCH, local trauma centers with neurosurgery; CTZ, local trauma centers without neurosurgery; PST, trauma emergency rooms.

Figure 5 shows the trend of the PEMAAF score in the four different levels of trauma care during morning shift and during nighttime. Hospitals within the CTS and CTZNCH reported a high PEMAAF score (> 10), which would enable for full activation of their hospital MCI plan during the simulated MCI. Hospitals classified as CTZ and PST reported lower levels of PEMAAF score (< 10) in the four simulations, thus showing a more conservative approach. Of note, no major changes in the different time slots and during holidays were observed.

Figure 5. PEMAAF Score Detailed Trends According to Morning and Night Shifts (Seasonal).

Abbreviations: PEMAAF, Proximity, Event, Multitude, Overcrowding, Temporary Ward Reduction Capacity, Time Shift Slot [Prossimità, Evento, Moltitudine, Affollamento, Accorpamento, Fascia Oraria]; CTS, specialized trauma center; CTZNCH, local trauma centers with neurosurgery; CTZ, local trauma centers without neurosurgery; PST, trauma emergency rooms.

Specificity and sensitivity of the PEMAAF score model are illustrated in Figure 6. A PEMAAF score of 11 was associated with a sensitivity of 86% and specificity of 63%, while a PEMAAF score of 12 was associated with a sensitivity of 58.5 % and specificity of 86%. Therefore, a PEMAAF of 11 favored a system that functions at a higher sensitivity, by alerting hospitals that might not receive any casualty, thus encouraging a prudential decision that is made in the general interest but that could negatively impact the planned activities. On the contrary, a PEMAAF of 12 favored a system that functions at a higher specificity, by activating few hospitals MCI plans, thus committing less resources but also incurring in the risk of a sub-activation of the response.

Figure 6. Specificity and Sensitivity of the PEMAAF Score Model.

Abbreviation: PEMAAF, Proximity, Event, Multitude, Overcrowding, Temporary Ward Reduction Capacity, Time Shift Slot [Prossimità, Evento, Moltitudine, Affollamento, Accorpamento, Fascia Oraria].

Discussion

This manuscript provides a comprehensive description of the multi-step approach used to design, test, and validate a new tool to support emergency medical systems during MCIs, by predicting the required level of hospital MCI plan activation after an initial assessment. Indeed, the PEMAAF score proposed in the Metropolitan City of Milan provides a snapshot of key data and characteristics of an MCI, thus estimating the adequate hospital response level and therefore supporting hospital managers, incident commanders, and hospital medical responders in the decision to appropriately activate the hospital MCI plan. The retrospective analyses clearly demonstrated how the use of the PEMAAF score would have correctly reduced the activation of the hospital MCI plan according to the real need dictated by MCIs that occurred in Milan. The potential of the tool has then been further emphasized by the simulations performed in the months of August and December, which represent the most challenging and busiest months in Milan.

By including a series of indicators (Proximity, Event, Multitude, Overcrowding, Temporary Ward Reduction Capacity, and Time Shift Slot) widely identified in the literature as crucial to determine the effectiveness of the re-organization procedure following an MCI, ^3,17–19 the PEMAAF score also allows to avoid unjustified activations of the hospital MCI plan, thus safeguarding elective and routinary hospital activities, thus not hampering the continuity of care and assistance. ¹¹

The PEMAAF score enabled the MCI management within the Metropolitan City of Milan to reach a much higher plateau of innovativeness, by enhancing communication and response strategies between the DC and the regional trauma network, through the establishment of a common language and shared operating procedure. Remarkably, the application of the PEMAAF score allows hospitals to act as a network, in which the hospital MCI response is no longer managed by very single hospital as standalone unit, rather guided in a coordinated manner. The importance of a standardized and coordinated response to MCIs, based on shared terminology and common understanding, has already been highlighted in available peer-review literature. ^17,20–24

Additionally, the implementation of the PEMAAF score enabled coordination centers and DCs to provide for a hub and a capacity for the hospital network, and to contain the compressive effect on ordinary activities and flows to the hospital EDRs on the territory, thus protecting the need for the non-MCI health, which is incessant by definition. ^11,25

Yet, the implementation of such a tool requires the continuous training of all operators involved in the MCI response, especially given the current limited knowledge in terms of MCI preparedness and response in the Italian context. ²⁶ To this regard, simulations and exercises can effectively foster this awareness, ²⁷ ensuring that the operators’ training and their confidence is constantly monitored and that any shortcomings are identified and contained. ⁹

Finally, it should be emphasized that the multi-step methodology adopted to design the PEMAAF score followed a sound scientific approach based on an existing framework ^14–28 and relied on the participation of all the local stakeholders (regional trauma network and the DC), which have been involved from the very first step of the process and actively contributed to refine and validate it. Moreover, incorporation of validated indicators allows its application to a broader scale (eg, at a regional or national level). ²⁸ As such, the tool has also the potential to be adapted and implemented in different contexts than where it has been conceived and developed. ^29,30

Limitations

Since the study was conducted in an Italian urban area, result validity needs to be verified in other environments.

Additionally, artificial intelligence algorithms were not incorporated, which have the potential to enhance the accuracy of the scoring system.

It is worth mentioning that the PEMAAF score has never been tested through a blind exercise (ie, in the case where hospitals had not been warned); such an event was expected to be carried out in the first months of 2020, and it had then been canceled due to COVID-19.

Conclusions

This manuscript offers a comprehensive overview of the multi-step process employed for developing, testing, and validating a novel tool designed to assist the emergency medical system during MCIs. The tool predicts the necessary activation level of hospital MCI plan, following an initial assessment.

Specifically, the PEMAAF score, tested in the Metropolitan City of Milan, presents crucial data and MCI characteristics, enabling the estimation of an appropriate hospital response level.

This, in turn, aids hospital administrators, incident commanders, and medical responders in making informed decisions regarding the activation of the hospital’s MCI plan.

The PEMAAF score also serves the purpose of preventing unwarranted activations of the hospital MCI plan. Consequently, it helps to safeguard regular hospital activities, ensuring the uninterrupted delivery of care and assistance.