Development and Psychometric Properties of Decision-Making Scale for Emergency Hospital Evacuation in Disasters

Abstract

Background:

The evidence shows that the need for emergency evacuation in hospitals has arisen. Designing an emergency evacuation decision making tool increases the confidence of hospital managers in the decision made. Therefore, this study was aimed at the development, and the psychometric properties, of the decision-making scale for emergency hospital evacuation in disasters.

Methods:

This study was done in 2 phases of qualitative study and literature review and designing and psychometric properties of the instrument. After development of the primary item pool, the psychometric properties of the questionnaire were evaluated. In this regard, face and content validity, internal consistency (Alpha’s Cronbach), reliability (ICC), and stability were assessed.

Results:

In the validity stage of the instrument, 4 items were removed. Also, 4 items were modified and 2 items were merged. The number of items was thus decreased to 64. After CVI calculation, 5 items were removed, 4 items were modified, and 2 items were merged. As a result of this, the number of items decreased to 58 items. The scale has good reliability and stability.

Conclusion:

It seems that the instrument could be useful in decision-making for emergency hospital evacuation in disasters.

Introduction

Among the organizations involved in disaster management, the health system has a specific place since health is the first and foremost demand of the people, particularly after disasters. ¹ The role of hospitals, and other healthcare centers, is critical in a crisis because they play a chief role in managing and controlling the consequences of such situations. ^2,3 Hospitals are crucial elements in creating emergency preparedness in most countries and must be fully functional during disasters. ⁴ The hospital’s vulnerability to the consequences of catastrophes interferes with providing healthcare services for the community. ⁵

Numerous studies have indicated the need for emergency hospital evacuation (EHE) due to fire, hydro-meteorological hazards, terrorist threats, and other natural disasters. On the other hand, EHE is a complicated process because of patients’ constant need for care, mobility problems, transportation problems, and understanding of the need for evacuation. ^6–8 Nowadays, hospitals in most countries, even developed ones, are not yet sufficiently prepared for a successful emergency evacuation. ^9–11 Over the past 20 years, more than 100 hospitals and 65 healthcare centers have been destroyed or severely damaged in Iran, thus requiring emergency evacuation due to natural hazards. A 2014 safety assessment report from 224 hospitals in Iran demonstrated that most of them (54.5%) were in the high vulnerability range. ¹² Disaster risk reduction plans have always been a challenging issue in the Iranian healthcare system. Despite the achievements and invaluable actions taken in disasters such as the Bam earthquake, Tropical Cyclone Gonu, and the extensive efforts to provide infrastructure, the Ministry of Health still requires fundamental measures to improve the management system and reduce the risk of disasters. ¹³

Due to the increase in artificial and natural disasters in recent years, we must plan to better respond to these hazards. ¹⁴ Making decisions for an EHE is a far more complicated process for hospital managers because the process is generally complex and becomes more difficult in a crisis. ¹⁵ It results in the loss of financial and human resources, and aggravates the medical problems of the affected area. On the other hand, failure to evacuate in serious hazards can result in mortality and aggravation of the hospitalized patients’ conditions. Hence, proper decision-making is crucial in an EHE. ¹⁶ There are several variables in an EHE decision-making. These, including variables such as receiving accurate information about the threat, backup issues, patient-staff-related outcomes, assessing the treatment needs of local people, and the effect of EHE on community resilience, are interfering factors in the EHE decision-making process. ¹⁶ The history of EHE shows that uncertainty affects all aspects of EHE decision-making. ^17,18 Better organization of the resources leads to the success of EHE. ^19–22 The decision to evacuate a hospital in an emergency is not easy, but it must be instantly made. ^11,23–25 It is an important risk management tool and should be performed whenever patients and staff are at risk. ²⁶

Despite the importance of timely decision-making for EHE, few studies have worked on this issue. Studies related to the EHE decisions in disasters have highlighted the urgent need for research. Although there is a crucial need for decision-making instruments for EHE, there is no reliable instrument. Developing a decision-making instrument for EHE in disasters helps the hospital managers quickly and accurately evaluate the situation. The managers can make a proper decision about EHE by assessing the effect of the accident on the performance and ability of the hospital to continue providing services. Correspondingly, access to such instruments reduces the error rate in decision-making and increases the manager’s power in disaster risk management. Therefore, this study was aimed at developing and deciding the psychometric properties of the decision-making scale for EHE in disasters.

Method

This exploratory sequential study includes a qualitative approach with quantitative data collection for developing and analyzing the psychometrics of an instrument. ^27,28 Therefore, we used a combination of the data in the instrument-making stage and data interpretation. In other words, we developed the instrument according to the extracted concepts, themes, categories, subcategories, codes, and semantic units in the qualitative stage. In the end, we analyzed the results of the qualitative and quantitative parts of the research together.

Generating the questionnaire items

This study employed a deductive-inductive approach to generate the items of the instrument. 1 of the advantages of using this approach is exploiting the available literature and other questionnaires. This helps the researchers to multi-dimensionally cover the topic. The research team developed the initial instrument according to the systematic review section, qualitative interviews, themes, categories, subcategories, and semantic units obtained. They defined the decision-making for EHE in disaster based on the findings and results of the qualitative content analysis, the constructs, and the sub-constructs (subscales). Then, they generated the initial items according to the extracted definitions, dimensions, and components from the content analysis and review study. They used a deductive-inductive approach to create the instrument items. The description was explained based on a review study, and most of the items were made based on the extracted categories in the deductive approach. On the other hand, they inductively made the other items and formed the initial pool of items, thus employing a qualitative interview.

Psychometric analysis of the scale

Determining the face validity

First, the researcher qualitatively evaluated the scale for face validity. To do so, 15 experts in disaster evaluated the instrument’s dimensions and their relationship. The researchers revised the items based on the comments provided by the experts. To determine the face validity by quantitative method, 10 members of the Disaster Risk Management Committee affiliated with The Tehran University of Medical Sciences examined and modified the instrument concerning the difficulty level, degree of incompatibility, and ambiguity. The research team used Item Impact Scores to evaluate the face validity of the instrument quantitatively. In other words, 10 committee members determined the importance of each item of the scale on a 5-point Likert scale from 1 (not necessary) to 5 (very important). Then, the impact score was calculated based on the following equation:

$$\rm Impact \ score = Significance \times Frequency \ (Percentage)$$

The impact score of each item should not be less than 1.5, i.e., the face validity of the items with an impact score of higher than 1.5 is acceptable.

Determining the content validity

The content validity index (CVI) and content validity ratio (CVR) were used to evaluate the instrument’s content validity. The initial instrument and the required criteria were emailed to 30 experts in the field. After collecting their responses, the CVI and CVR indices for each item were calculated; if the item didn’t get the appropriate score, it was omitted from the instrument.

Content validity index

We employed Waltz and Bausell’s approach for evaluating the content validity index. ^29,30 Therefore, the experts assessed each item’s relevance, clarity, and ease on the 4-point Likert scale ranging from ‘not at all’ to ‘ great extent.’ Then, the CVI of the instrument was calculated based on the following equation:

$${\rm{CVI}} = \displaystyle{{nE} \over N}$$

nE: The number of panelists rating 3 and 4.

N: Total number of panelists

The minimum acceptable value for CVI was 0.79; if it was less than 0.79, that item was removed from the instrument. ³⁰

Content validity ratio

We explained the purpose of the instrument to a panel of experts and asked them to evaluate the items according to their necessity. They categorized each into 2 groups of necessary and unnecessary items. Then, the researcher calculated the CVR index using the following equation:

$${\rm{CVR}} = \displaystyle{{nE - N/2} \over {N/2}}$$

nE: Number of panelists who selected the necessary option

N: Total number of the panelists

The minimum acceptable CVR value is determined based on the number of experts who evaluated the items (Table 2). We omitted the items with a CVR value of less than the desired value from the instrument. Since there were 30 experts taking part at this stage, we excluded the items with a CVR value of less than 0.33.

Reliability

We assessed the reliability of the scale using internal consistency. ³¹ To evaluate the instrument’s internal consistency, 290 managers of Disaster Risk Management Committees of hospitals affiliated with The Tehran University of Medical Sciences and Mazandaran University of Medical Sciences completed the scale. The results revealed that Cronbach’s alpha correlation coefficient was 0.7, showing a satisfactory internal consistency. Moreover, the scale’s reliability was assessed using a test-retest approach with a 2-week interval, according to Waltz et al. ³² To conduct the test-retest, 50 Disaster Risk Management Committee members completed the MWWFCQ twice with a 2-week interval. The researchers considered several facts while doing the sampling. First, they paid attention to the missed items: if the participants didn’t answer an item, they asked them to do it. Also, they evaluated the stability of conditions in the test-retest stage. In order to do this, participants were asked, while completing the questionnaire for the second time, if they had ever attended an EHE-related training course or practiced it. If they had attended such classes or practiced EHE, they were excluded from the study. After collating both datasets, the Intra-class Correlation Coefficient (ICC) was calculated for the 3 subcategories and the whole scale. This test is the ratio of intergroup variance to the total variance. The ICC value of 0.8 and higher shows satisfactory stability between the 2 tests. ³³

Item weighting

There are several ways to score an instrument. In this study, regarding the nature of the tool, the viewpoints of disaster risk management experts were benefiited in order to weigh the items. ³⁴ Ten experts scored the items based on the importance and effect of the item on decision-making for EHE in a 5-point Likert scale ranging from not important to very important. Then, the researcher calculated the mean scores for each item as “not important = 1, slightly important = 2, moderately important = 3, important = 4, and very important = 5”. Then, according to the experts’ opinions, the weighting average was estimated for each item.

Scale Scoring

First, in scoring the scale as the study tool, the researcher calculated the weight of each item and multiplied it by the numerical value of the response option and the total score of the tool was estimated this way. The researchers applied the mathematical logic of 33% to determine the cut point, for which the response was divided by 3. Ultimately, the final scale was determined as non-emergency evacuation, preparedness for emergency evacuation, and emergency evacuation.

Results

Items Generation

According to the systematic review and a qualitative study, the researchers generated the initial scale, including the EHE decisions factors. ³⁵ They combined the categories extracted from the systematic review and the qualitative interview and generated a pool of items (Table 1).

Table 1. Initial estimate and number of items suggested by the instrument

Psychometric Analysis of the Instrument

According to comments from the experts and research team, the researcher removed 4 items and modified 3. As a result, the number of items decreased to 64 (Table 1).

Face validity

Results of the impact score showed that 4 items (items 2, 8, 9, and 10) had an impact score of less than 1.5. However, items which scored less than 1.5 were not omitted from the questionnaire at this stage. We removed or modified them according to the CVR value.

Content validity

Content validity was evaluated qualitatively and quantitatively. The 64-item scale was sent to 15 experts in qualitative content validity and they commented on the items’ appropriate word use and ease. Among them, 10 experts (0.66%) completed evaluating the scale, resulting in changes and modifications in several items.

Content validity ratio

After calculating the CVR, a decision was made to preserve or remove items according to the comments of the target group, experts, and the research team.

Content validity index

To accomplish CVI, 15 experts determined the relationship between scale items according to its sub-scales in the 5-point Likert scale. CVI was calculated for items which scored 3 or 4 (highest score). When there are 15 panelists, the minimum numerical value of the CVI is 0.75 with a P -value of 0.05 according to the Lean table.

After CVI calculation, those items with numerical values between 0.70 – 0.79 were considered debatable items, and those with a numerical value lower than 0.70 were unacceptable. As a result, the researcher removed 5 items (items 2, 8, 9, and 10), modified 4 (items 5, 6, 7, and 8), and merged 2. The number of items therefore decreased to 58 items (Table 2).

Table 2. CVR and CVI values of the items of ‘EHE decision-making in disasters’

Reliability

Internal consistency

Cronbach’s alpha coefficient was calculated to determine internal consistency. If the Cronbach’s alpha value is higher than 0.7, the instrument will have appropriate internal consistency. ²⁹ To do so, 290 managers and members of the Disaster Risk Management Committee completed the scale. The Cronbach’s alpha coefficient for this study was 0.738, showing that the scale has good internal consistency (Table 3).

Table 3. Cronbach’s alpha coefficient for the 3 dimensions of the instrument

Stability

To determine the scale’s stability, the researcher asked 10 managers of Disaster Risk Management Committees in hospitals affiliated with Tehran University of Medical Sciences to complete the instrument twice, with a 2-week interval. Thus, by comparing the responses through retesting, the managers calculated the Pearson correlation coefficient. If the reliability coefficient between the 2 tests is more than 0.7, the stability of the questionnaire is acceptable. This scale enjoys high stability (Pearson correlation coefficient: 0.888; P-value: 0.000; Number of samples: 10).

Interrater reliability

To calculate reliability in the second phase in 20 hospitals affiliated with Tehran University of Medical Sciences, 2 members of the Disaster Risk Management Committee independently completed the decision-making scale of EHE in disasters. The researcher analyzed the data using a weighted kappa statistic and interpreted the results according to the instructions of Cicchetti and Sparrow (1981) and Fleiss (2011). ^36–38 Thus, values less than 40 to 59 were considered weak, between 60 and 74 were good, and higher than 74 were excellent (Table 4).

Table 4. Inter-rater reliability by items based on weighted kappa statistic

Weighing the Items of the Instrument

The researcher used the comments of 10 experts in disaster to weigh the items. They rated the effect of the items, based on their importance in the EHE decision, ranging from 1 to 5 with 1 for not important, 2 for slightly important, 3 for moderately important, 4 for important, and 5 for very important. Then, the researcher calculated the mean value for each item. According to the calculations and comments provided by the research team, the numbers related to the mean were rounded and determined to range from 1 to 3. Considering the answers to each item being of different values Since the answers to each item have different values, the researcher she multiplied the value of the item by its weight, added the scores of all the items, and calculated the final score of the scale. Table 5 illustrates the results related to the weighting of the items. In order to determine the cutting point of the instrument score, the 33% rule was used. (Table 5, 6).

Table 5. Weighting scores of tool items ‘Emergency hospital evacuation decision in response to accidents and disasters’

Table 6. Scoring and interpreting ‘Emergency hospital evacuation decision in disasters questionnaire’

Discussion

Overall, this study revealed that developing a decision-making scale helps hospital managers reduce their mental stress and legal pressure, a sentiment which was also highlighted by Voyer. ³⁰ He emphasized the need to develop planning and preparation processes for potential cases in full detail in the hospital. They believed that we should support the policy-makers and decision-makers with reliable instruments to reduce decision-making responsibility in crises and emergencies. ³⁹ Correspondingly, King highlights the need for access to a single form to collect information about accident conditions in the hospital. It leads to an increase in the efficiency and performance of managers in the decision-making process for EHE. ⁴⁰

Decision support systems operate as an integrated database which increase the power and ability of managers to analyze the situation accurately in order to make the right decisions in the crisis. ⁴¹ Despite the importance of this issue, there is no reliable scientific instrument concerning decision-making for EHE. This indicates the complexity of this concept. The difficulty and complexity of quantifying the influential factors and their relevance to the management conditions and the health system are the reasons for the lack of access to standard international instruments in this field. Koeing emphasizes that in deciding for EHE, 1 should identify the relevant and practical factors through research. In other words, evidence-based performance should also be considered in designing the international EHE decision-making guidelines. ⁴²

There were 14 items in the scale for information about assessing the risk and life-threatening factors as related to population density, hospital features, and accident dimensions. An EHE command is given based on the threat of a situation. Hence, hospital authorities regularly assess the nature of such threats in relation to available resources, and determine the operating cycles. ⁴³ Risk assessment is very important for patients and staff. To avoid miscalculation, a multidisciplinary professional team should estimate the potential risks for patients and the hospital infrastructures based on reliable information. ^23,44

In 2013, Belflower conducted a qualitative interview with nursing home managers and revealed that risk evaluation is the main factor in rapid and accurate decision-making. ⁴⁵ The safety and health of patients and staff are essential aspects of the emergency evacuation process. ²⁶ In a 2009 study, Fennell and Levitan estimated the risk of storms. They emphasized that the threat to the safety of patients as a result of accidents is an essential factor in emergency evacuation decisions. ⁴⁶ Factors influencing the nature of the accident include time of the accident prediction, its severity, the affected area, and its duration. ⁴⁷ Emergency evacuation is a vital risk management tool, especially when patients and staff are at high risk. ²⁶ Perceived risk of the threat and a thorough risk analysis also influence emergency evacuation. Rega proposes that the potential effects of disasters in hospitals should be simulated before they occur so as to estimate threat level at the time of hazard, and to carry out careful planning. ⁴⁸ Designing and using data collection checklists is very helpful in quickly assessing the risk of disasters. It increases the accuracy of the EHE decision-making process.

The feasibility section for providing medical care services consists of 13 items, including 2 subsets of hospital vulnerability assessment and hospital capacity assessment. Hospital vulnerability varies from country to country and is based on geographical location and the event. More than 50% of healthcare centers are located in high-risk areas in some countries such as South America. ²³ While in the UK, 8% to 9% of healthcare centers are located in places with high risk. ⁴⁹

High-quality hospitals with many stories are standard in developed countries. However, there are large hospitals, which can be found in megacities in developing countries too. Emergency evacuation in 1-story hospitals have fewer problems than in multi-story buildings. ⁴ On the other hand, Vugrin emphasized that the decision to evacuate a hospital depends on the hospital’s ability to continue providing appropriate medical care for its patients in 2015. ⁵⁰ Similrly, Goetschius showed in his thesis that the decision to relocate or evacuate a hospital is essentially based on whether the staff and the center can continue providing standard patient care. ⁴⁷

Hospital vulnerability analysis is crucial in estimating the feasibility of continuing medical care. In 2013, Hasol stated that the decision for EHE requires considering several factors, including the vulnerability of critical infrastructure, electricity for supporting equipment, availability of the roads around the hospital, and having access to safe routes for transporting patients without elevators. ⁵¹ Assessment of hospital infrastructure should be done before disasters. Hence, decision-makers can assess the degree of vulnerability and the potential consequences of an impending disaster in the hospital building and its surrounding areas. ⁵² Continuous control of the accident and hospital conditions is essential in an emergency evacuation. ²⁴ In recent years, the issue of evaluating the structural and non-structural safety of hospitals has received much attention in Iran. ¹² Educating managers regarding the importance of focusing on the hospital’s vulnerability and capacity assessment after a disaster is crucial in emergency evacuation decisions, and is in line with implementing the 2015-2030 Sendai Framework for Disaster Risk Reduction.

Assessing the damage to critical hospital infrastructures is crucial. The maintenance of the hospital’s treatment activities depends on its infrastructure. Zane et al. ²⁵ considered the self-assessment of critical hospital infrastructures as essential to assist the decision-making team regarding the hospital’s ability to accommodate on-site. Moreover, they believe that estimating the required time for EHE is important in EHE decisions. ²⁵ The emergency hospital evacuation is a time consuming process and much more complex than other buildings. Hospitals are a collection of interconnected buildings, thus requiring specific solutions in a disaster. Furthermore, some patients may have mobility problems. ⁵³

The emergency evacuation prerequisite section of the decision-making scale has 30 items with 2 subsets of executive coordination and the possibility of emergency evacuation of patients. In the former, organizational and regional officials’ legal delegation of decision-making is evaluated. The decision to evacuate a hospital entails potential legal liability, financial issues, and political considerations. ⁵⁴ Disasters can have political consequences therefore, the decision to evacuate a hospital is influenced by potential for negative political outcomes. ⁵⁵

Hershy emphasized that EHE-related legal challenges are the main concerns of hospital managers. ⁵⁶ Furthermore, EHE prerequisites affect the required time estimation. The number of ambulances, number of personnel needed, equipment required for patient transportation, the hospital’s internal and external communication system, capacity of the destination hospital, and patients’ evacuation routes are chief among the issues faced. In 2012, Berwari emphasized the evaluation of hospital management capability for rapid emergency evacuation. ⁵²

Several studies on EHE education have highlighted that problems such as support, equipment, human resources, information management, communication, and intra- and inter-organizational coordination influence the EHE process’s success. ^{5,16,23,51,57–59} Furthermore, number of patients, their current condition, medical care needs, mobility, number of available staff, availability of road and safe transportation, and availability of suitable and safe alternative accommodation are central issues affecting EHE decision-making. Similarly, Zaenger et al. have stated that number of patients, availability of equipment for transferring patients, communication system, and coordination with the authorities, all affect EHE decisions. ²⁴ In a 2008 review article on EHE experiences, Bagarai et al. stated in 2008 that the biggest challenges are internal communications (teledensity), lack of access to an elevator, limited resources, and the need for a memorandum with other hospitals for transporting patients. ⁶⁰

Considering internal and external factors affecting EHE implementation is necessary for decision-making on emergency evacuation. A successful EHE program depends on effective communication inside and outside the hospital. Coordination with other government agencies, in particular, should be achieved through their involvement in various planning, practice, and mutual memorandum. ⁵⁸ Communication and information management are crucial elements in chaotic situations. Pre-organized checklists and worksheets are beneficial in the field of communication. Augstin suggested in 2005 that Emergency Evacuation Packages containing worksheets, telephone numbers, handling equipment, and disposable items for patients should be prepared in the hospital in advance. ⁵⁷ It is essential to consider the hospital’s ability to safely evacuate patients in evacuation decisions. Managers and planners must estimate the required time for evacuating all patients from 1 hospital and their transmission to other hospitals. An efficient step in ensuring patients’ safety in the evacuation process is using resource control forms and prerequisites required for emergency evacuation.

Limitations

Lack of access to the full text of some articles, not using some databases, and studying the sources in English only in the systematic review are limitations of this study. Also, factor analysis was not possible in the psychometric stage due to the type of the items.

Conclusion

The present study results showed that the instrument enjoys good validity and reliability. It also showed that it can be integrated with the functional safety dimension of the Hospital Safety Index (HSI). The researchers recommend that the system designed in this study be used as an educational aid for members of the Hospital Disaster Management Committee to master emergency evacuation decision-making skills in disaster scenario-based exercises in Iran.