Hospital Performance Evaluation Checklist in Context of COVID-19 Pandemic: Design and Validation

Abstract

Objective:

Around the world, pandemics have been considered among the main hazards in the last 2 decades. Hospitals are 1 of the most important organizations responding to pandemics. The aim of this study was to design and develop a valid checklist for evaluating the hospitals’ performance in response to COVID-19 pandemic, for the first time.

Methods:

This study is a mixed method research design that began in February, 2020 and was conducted in 3 phases: Designing a conceptual model, designing a primary checklist structure, and checklist psychometric evaluation. Known-groups method has been used to evaluate construct validity. Two groups of hospitals were compared: group A (COVID-19 Hospitals) and group B (the other hospitals).

Results:

The checklist’s main structure was designed with 6 main domains, 23 sub-domains, and 152 items. The content validity ratio and index were 0.94 and 0.79 respectively. Eleven items were added, 106 items were removed, and 40 items were edited. Independent t-test showed a significant difference between the scores of the 2 groups of hospitals (P < 0.0001). Pearson correlation coefficient test also showed a high correlation between our checklist and the other. The internal consistency of the checklist was 0.98 according to Cronbach’s alpha test.

Conclusions:

Evaluating the hospitals’ performance and identifying their strengths and weaknesses, can help health system policymakers and hospital managers, and leads to improved performance in response to COVID-19.

In recent years, epidemics, and biological events have occurred more than ever. The incidence of epidemics and pandemics of emerging and re-emerging infectious diseases, as well as bioterrorism, is increasing. ^1,2 The epidemics of SARS, MERS, and COVID-19 had physical, and financial, as well as psychological, and social consequences. ^3–6 Hospitals and other health care facilities always play a critical role in response to disasters, which becomes much more important in epidemics/ pandemics. It is therefore necessary to have preparedness plans for such biological disasters, as well as having a proper performance during the response phase. ^7,8

One of the greatest epidemics of the last several decades was COVID-19, which started in China (December 2019), rapidly became a pandemic, and caused a sharp rise in hospital admissions. ⁹ Hospitals are 1 of the most responsive centers with respect to COVID-19. Since SARS-CoV-2 virus and the resultant disease are still complex, the response dimensions of hospitals have been greatly complicated during this disaster. Proper hospital performance in COVID-19 management should be a multidimensional approach having high accuracy and continuity. It should be continuously evaluated. The evaluation would identify the strengths and weaknesses of hospitals, determine the response elements, and distinguish appropriate strategies to improve their performance. ¹⁰

Regarding risk management of biological events, various studies have been conducted in each phase of a disaster (prevention and mitigation, preparedness, response, and recovery). ^8,11–13 Most studies have been conducted on influenza, ^14–17 Ebola, ^1,18 SARS, ¹⁹ and man-made events such as bioterrorism, ^20,21 as well as other epidemics. ^13,22 In most studies, the important role of hospitals in management process and response to epidemics has been addressed. Since the beginning of COVID-19, several researchers have studied clinical features, diagnosis, and treatment, as well as clinical and economic consequences of the disease. ^9,10,23–25 Some have also studied preparedness, response, incidence commanding system, and allocated resources. ^26–31 A few studies have been conducted for instrumentation, to evaluate the hospitals’ performance during response to biological events; these studies have only been related to the design of checklists for assessment of hospitals’ preparedness. ^1,20,32 Now that health systems are involved in COVID-19 pandemic, the need for performance evaluation of such organizations would be valued more than their preparedness; which is a matter of pre-disaster situation. On the other hand, in order to properly evaluate the hospitals’ performance during response to COVID-19 pandemic, access to a valid checklist would be essential. Such a tool would provide a platform that would enable hospitals to evaluate their response to COVID-19, and play their role better.

The aim of this study was to design a checklist to evaluate various dimensions of hospitals’ performance, and provide guidance for preparedness measures, identifying strengths, and weaknesses, as well as prioritizing actions in response to this disaster.

Methods

This article is extracted from methodological research. The aim of this study was to design a valid checklist to evaluate the hospital performance in COVID-19 pandemic. The study started in February 2020 and was designed and implemented in 3 main phases (Figure 1).

Figure 1. Workflow diagram.

In the first phase, databases, and search engines like PubMed, Scopus, and Google Scholar, as well as Web of Science were searched. Relevant articles, guidelines, and checklists on hospital preparedness, and performance for biological events such as epidemics of influenza, Ebola, SARS, and COVID-19, as well as bioterrorism events were found and studied. Selected keywords included hospital, preparedness, epidemic, and pandemic, as well as checklist, evaluation, performance, and COVID-19. After reviewing the articles, guidelines, and checklists, a basic conceptual model was designed (Figure 2).

Figure 2. PRISMA diagram of study included in the systematic review.

In the second phase, the deductive-inductive method was used to extract the items of the basic conceptual model. Then an initial pool of items was formed. This phase consisted of 2 parts: (1) Scoping review of literature and relevant checklists, (2) Launch of an expert panel including 8 subject matter experts (Table 1). Finally, the primary checklist was developed.

Table 1. Participants’ expertise in different phases of the research

In the last phase, the checklist psychometrics were carried out using the quantitative and qualitative methods as mentioned below.

Face Validity

A qualitative method was used to determine the face validity of the tool. The primary checklist was presented to 30 participants including disaster management specialists (as experts), and members of the COVID-19 hospital Disaster Risk Management Committee as checklist users (Table 1). Participants’ criteria for decision was the ambiguity, difficulty, suitability, and structure, as well as checklist organization in the form of domains, subdomains, and clarity. Based on the comments suggested by the participants (Table 2), some items were removed, and some were edited.

Table 2. Relevant checklists of preparedness or performance in biological disasters

Content Validity

Qualitative and quantitative methods were used to determine the content validity of the tool. Fifteen specialists with varied expertise took part in this exercise. They were required to have expertise in hospital disaster management, adequate familiarity with the health system, and/ or adequate knowledge of biological events. The participants’ expertise is shown in Table 1. Participants’ opinions about the checklist content were collected and summarized. Then necessary corrections were made on items. A few items were removed, some new items were added, and some were revised.

Content validity of the checklist was measured quantitatively in two parts: (1) Content validity ratio for the necessity of items selected, (2) Content validity index to ensure the optimal design of items in content measurement. Experts’ opinions were collected based on a 3-choice scale (essential, useful but not essential, and not necessary). Participants’ responses were computed based on the formula: CVR = (n _e -N/2)/(N/2), where CVR = content validity ratio, n _e = number of subject matter expert raters indicating “essential,” N = total number of subject matter expert raters, and the content validity ratio of each item was determined. According to Lawshe table, ³³ the items which got a score higher than 0.56 were accepted, and other items were removed. In order to determine the content validity index (CVI), the experts’ opinions were reviewed for relevancy of each item based on a 4-point scale. The options for relevancy were: Highly relevant, quite relevant, somewhat relevant, and not relevant. I-CVI (Evaluation of content validity at the level of each item in the checklist) and S-CVI/ Ave (Average of I-CVI) was computed based on the following formula for each item considering average approach. ³⁴

$${{\rm{I - CVI/ Ave}} = {{\left( {{\rm{the\;specialists\;who\;have\;scored\;this\;item\;}}3{\rm{\;and\;}}4} \right)} \over {{\rm{the\;total\;number\;of\;experts}}}}}$$

At the end, the items with a score higher than 0.78 did not change, and items with a score less than 0.78 were removed.

Construct Validity

To determine the Construct validity, the methods of known-groups and convergent validity were used. Known-groups is 1 of the methods used to support the Construct validity. It is presented when a test can discriminate between 2 groups known to differ on the variable of interest. ^35,36 The performance scores of the 2 groups of hospitals were compared (group A: COVID-19 admitting hospitals, and group B: other hospitals that did not meet the care conditions of COVID-19 patients). We expected higher scores from Group A.

A rater team was appointed for each group (Table 1). Pre-requisites for selecting raters included: employment in the hospital, knowledge about the hospital, and membership of a Risk Management Committee/Hospital Disaster Management Team of the Hospital. The checklist was provided to both rater teams and the hospitals’ performance was evaluated. The results were analyzed by independent t-test and Item-Total correlation (SPSS 26, IBM Corp., Armonk, NY, USA).

Convergent validity is another method used to evaluate Construct validity. ^35,36 This method refers to the degree of relationship between 2 structural criteria that should be theoretically related to each other. While there were several similar checklists available that were intended to be used in this study (such as the CDC Coronavirus checklist, WHO Ebola Virus checklist, WHO Influenza Virus checklist, AHA Biological and Chemical Terrorism Preparedness checklist ^37–40 ), the checklist provided by the Ministry of Health of Iran was utilized for comparison in the known groups method ⁴¹ . Our checklist plus another similar 1 (an Iranian checklist including 90 items to evaluate hospitals’ preparedness during the COVID-19 pandemic), ³⁷ were provided to raters of group A. Both checklists used a 3 point scale measurement. The degree of relevance of their evaluation results was determined by Pearson correlation coefficient. It was expected that obtained scores from the 2 checklists would be positively correlated with each other at this stage. Raters’ qualitative feedback were also collected and analyzed.

Reliability

The reliability of the checklist was determined by internal consistency and Cronbach’s alpha, based on the results of group A rater (50 samples).

Participants

Participants with different specialties took part in this study, as shown in Table 1.

Results

The main purpose of this study was to design and validate a hospital performance evaluation checklist in response to COVID-19 pandemic. In the first phase, a search was done to find checklists relevant to evaluation of preparedness or performance in biological disasters. Six related tools were found. Their specifications and measurement dimensions are shown in Table 2 and all the tools were used to evaluate hospitals’ preparedness during the pandemic.

Previous tools and guidelines had studied preparedness, but the subject of the present research was performance checklist, so the conceptual model of the tool was based on the study of factors affecting the performance of hospitals in biological events, ⁸ and was developed according to the performance evaluation during the pandemic (Figure 3). The main domains of the checklist were determined as: risk management and planning, coordination and communication, infection prevention, and control, as well as diagnosis and treatment, training and exercises, and resources management.

Figure 3. The conceptual model.

In the second phase, the initial pool of items was formed based on the conceptual model and according to the following: (a) International guidelines and checklists related to evaluating the hospitals’ performance and preparedness in epidemics and pandemics, (b) Expert panel, (c) Related articles focusing on risk management of epidemics and pandemics. The items of the selected checklists were reviewed and compared with the items obtained from literature review and experts’ panel. In total, 247 initial items were obtained and placed in the checklist structure. Thus, the primary checklist was designed with 6 main domains, 23 subdomains, and 247 items. The checklist scoring scale was determined as a 3-choice scale; positive (3 points), borderline (2 points), and negative (1 point).

In the third phase, the psychometrics of the primary checklist was performed by quantitative and qualitative methods as mentioned below.

Face Validity

Regarding the results of qualitative face validity, 16 items were removed, and 15 items were revised in terms of grammar, ambiguity, and proper placement (under appropriate main domains and subdomains).

Content Validity

By qualitative content validity and after applying the experts’ opinions, 38 items were removed, 9 new items were added, and 22 items were revised. Added items covered the following matters: hospital safety index, specimen management in laboratory, corrective measures in laboratory, and spill management in the laboratory, as well as clinical ethics, resolving patient dissatisfaction, staff motivation, and having ICUs, as well as standard isolated rooms.

Results of quantitative content validity showed CVR = 0.79 and S-CVI/ Ave = 0.95. It was also noticed that CVR min = 0.23, CVR max = 1.00, and CVR of 45 items was less than 0.56. The analysis showed that I- CVI min = 0.69, I- CVI max = 1.00, and I- CVI of 10 items was less than 0.78 (Table 3). In this phase, 52 items were removed, 3 items were revised, and the secondary checklist was constructed including 150 items at this part (Table 7).

Table 3. Quantitative content validity

* I-CVI (item-level content validity index;

** S-CVI/Ave (scale-level content validity index, averaging method).

Construct Validity

To determine the construct validity, the known-groups method and convergent validity were used. In the comparison section of the known-groups method, 100 raters used our checklist for evaluation of groups A and B hospitals. Group A hospitals got the higher score. The mean and standard deviation of both groups are shown in Table 4. Comparison of the mean scores by independent t-test confirmed the hypothesis of the difference between the 2 groups (P < 0.0001, t = 9.05). So, the present checklist would distinguish the performance of 2 types of hospitals and has a suitable construct validity.

Table 4. Comparison of mean scores between 2 groups of hospitals

* COVID-19 admitting hospitals;

** Non-COVID-19 admitting hospitals; independent t-test, P < 0.0001, t = 9.05.

To determine convergent validity, we compared the checklist with a similar 1; an Iranian checklist including 90 items to evaluate hospitals’ preparedness in COVID-19 pandemic. ³⁷ Both tools were provided to raters of group A (50 participants) to evaluate the hospitals’ performance. Convergence between the results of both checklists was analyzed by Pearson correlation coefficient (Table 5).

Table 5. Results of correlation coefficient evaluation of 2 checklists

** Correlation is significant at the 0.01 level (2-tailed).

There was a high correlation between the scores of the 2 checklists (R = 0.93, P < 0.0001).

Reliability

Cronbach’s alpha was 0.98, indicating good internal consistency of the present checklist. All items were higher than 0.3, based on Item-Total correlation test (P < 0.0001), except items 29 (0.20) and 30 (0.21).

In brief, the conceptual model of the checklist was designed including 6 main domains and 23 subdomains (Figure 2). Necessary changes were made by review of literature and similar checklists, and interviews with disaster management specialists and hospital officials, regarding the specific circumstances of COVID-19 pandemic. The changes included: (1) integration of domains: planning with incidence command system, communication with coordination, diagnosis with treatment, and (2) determination of infection prevention and control as an independent domain. Finally, the considered main domains are; (1) risk management and planning, (2) coordination and communication, (3) infection prevention and control, (4) diagnosis and treatment, (5) training and exercises, and (6) resources management. The main purposes of these changes were to adapt the checklist to COVID-19 pandemic situation, and to make the tool easier to use.

Scoring

Considering the 3- choice scale (completed, in progress, not started) for the tool and its normative nature, the linear conversion method (based on the number 100) was chosen as a guide for evaluating scores and judging the checklist performance. The standard performance score of hospitals was calculated by the formula:

$$\rm LT = (NH-N_{min})/(N_{max} - N_{min}) x 100$$

Where NH = The raw score of the hospital performance, N_min = The minimum checklist score,

N_max = The maximum checklist score, and LT = The standardized score of hospital performance.

The following 4- part scale was selected for judging hospitals’ performance: “poor (0 - 24.9), moderate (25 - 49.9), good (50 - 74.9) and excellent (75 – 100).”

Field Feedback

According to qualitative feedback of the checklist field test (100 raters in 15 hospitals), 2 more items were added regarding recovery index of patients and the quality of respiratory care of patients.

Totally, 11 items were added, 106 items were removed, and 40 items were revised. All changes are shown in Table 6. Table 7 indicates details of the final checklist.

Table 6. Checklist changes after face validity and content validity

* QCV, Qualitative Content Validity;

** CVR, Content Validity Ratio;

*** CVI, Content Validity Index.

Discussion

The main purpose of this study was to design and validate a hospital performance evaluation checklist in response to COVID-19 pandemic. Although our team did not find a uniform model, and other studies had used variety of approaches, ^38,39 we attempted a comprehensive approach in 3 main phases; conceptual model development, primary checklist design, and checklist psychometrics.

We used methods of reviewing the literature and published guidelines, revision of similar checklists, establishing an experts panel, plus quantitative and qualitative statistical methods. Other researchers had used some of the methods such as reviewing the literature and published guidelines, ^38–42 interview with experts and establishing an experts panel, ^43–47 Delphi, ^21,42,48 or event and task analysis. ^42,45 This research used most of these methods in order to acquire the maximum validity. Brunsveld-Reinders et al., ⁴⁵ selected a primary model for development of a safety checklist for transfer of critically ill patients in hospitals, which is in line with part of our approach. The performance evaluation model of hospitals in response to biological events was used as a baseline model, ⁸ and attempts were made to accommodate this model with COVID-19 pandemic.

The present checklist has included subdomains of concurrent emergencies, clinical ethics, continuity of services, and psychological services, as well as patients and visitors training. These are considered as important subdomains that other tools have not addressed completely. ^{11,32,45–48} Field evaluation showed our checklist was replicable and feasible. Considering Table 8, the present tool is more complete than other checklists in some subdomains. ^{11,32,45–48} The present tool is more complete than CDC COVID-19 checklist in terms of having subdomains of clinical ethics, psychological services, concurrent emergencies, and inter- or intra-sectorial coordination, as well as decontamination, triage in biological events, rapid identification, and physical/financial resources. ⁴⁵ It is also more complete than WHO COVID-19 checklist in terms of having subdomains of concurrent emergencies, diagnosis, triage in biological events, and clinical ethics, visitors’ (patients and clients) training, logistics, and supplies, as well as physical resources. ³²

Table 7. Details of final checklist

Table 8. Comparison of domains and sub-domains of research checklist with other checklists

*Content overlap, Ep.W, Epidemic WHO; C.W, Covid19 WHO; I.W, Influenza WHO; E.W, Ebola WHO, B.A, Bioterrorism AHA; C.C, Covid19.

The distinctive feature of our checklist from other available tools is the consideration of the issue of clinical ethics. Although ethics is 1 of the challenges in pandemics and emerging diseases, ⁴⁶ other tools had not considered such domain in preparedness evaluation in COVID-19 or other biological events such as influenza, Ebola, and bioterrorism. In all epidemics, especially emerging diseases, it is possible to prescribe a combination of experimental drugs, and new drugs, as well as other complementary therapies. Inequality also may occur among patients. ^47,48

According to literature, all other tools were designed to assess hospital preparedness. The nature of the present tool is different from the others, as it has been developed to evaluate hospital performance.

The initial 247 items were reduced to 152 items (106 items were removed, 11 new items were added, and 49 items were modified and revised).

Various studies had different approaches to validate their checklists. This study applied quantitative and qualitative statistical methods to ensure the validity and reliability of the present checklist. Some studies did not use statistical methods such as Brunsveld-Reinders et al., ⁴⁵ Olivieri et al., ²¹ and Verdaasdonk. ⁴⁰ However, some researchers used quantitative and qualitative statistical methods like Burian et al., ⁴³ and Schmutz et al. ³⁸

Brunsveld-Reinders et al. ⁴⁵ and Hales et al. ⁴⁹ assessed the applicability of their tools by field evaluation and feedback. Our team also followed the same approach. Various methods have been used for validation and reliability of tools which might be due to different purposes, financial and time constraints, and application variety. In our study, the opinions of health system policy makers and hospital managers were applied through several interviews and experts’ panel.

The results of field evaluation showed that this checklist has appropriate construct validity and reliability. Since the average scores of group A hospitals were higher than group B, and there was a high correlation between the results of this checklist and similar tools, it can be concluded that the designed checklist is sensitive to differences and has good measurement capability.

Preparedness evaluation should be done in the preparedness phase of a disaster cycle; before disaster occurrence, so prevention and risk reduction measures are taken initially, and in order to reduce the effects of residual risks, preparedness measures would take place including planning, equipping, exercise, training, and designing warning systems. When a disaster occurs, such as COVID-19 pandemic in this particular case, there is a need to monitor, assess, and evaluate the response in order to identify performance weaknesses and take effective actions. Although preparedness evaluation might cover performance evaluation to some extent, it cannot evaluate all aspects of performance: (1) The availability of personal protective equipment is a case of preparedness, but the existence of this equipment and its proper use would be specified in the response phase only; (2) Being prepared does not necessarily guarantee a proper response. Only when disasters occur, would the effectiveness of measures be realized.

Standard performance checklists provide hospitals a uniform and valid tool, so they can be used: (1) to evaluate the current situation, (2) to determine the required resources, functional, and plan deficiencies, and (3) to compare with themselves or other hospitals in a time process. Applying the present checklist would result and ensure timely change in management procedures and interventions for pandemic impact reduction.

Conclusion

Pandemics have been considered major threats to the international community. An effective response is needed to reduce casualties and financial losses, in addition to maintaining preparedness. The role of hospitals as 1 of the most important organizations responding to pandemics is very critical. Valid tools can be helpful in identifying strengths and weaknesses and facilitating subsequent planning for health system decision makers and hospital managers. The main purpose of this study was to design a standard and valid tool to evaluate hospital performance in response to COVID-19 pandemic. The present checklist is evidence-based with appropriate validity and reliability. It is proposed that other countries can evaluate and improve their hospital performance by applying this checklist to their health systems. The indicators provided in this tool can also be a guide in developing biological disaster risk management plans of international and national health systems.

Suggestions

It is suggested that this checklist be used as a basic model for evaluation and comparison of the hospital performance. Given the variable nature of SARS-CoV-2 virus and the consequent changes in treatments and related care, annual checklist revision, is also recommended.

Limitations

A major limitation of this work was not considering international participants’ opinions in various steps of validation due to the urgent need of hospitals for such a checklist during the COVID-19 pandemic. This was ameliorated somewhat by the study and review of several internationally published papers and tools. Due to the formative nature of the checklist, exploratory factor analysis method could not be used to evaluate its construct validity.

Strengths

One of the strengths of this checklist is its comprehensive psychometrics. Using key people with maximum variability in compiling the checklist is another advantage. Evaluation of 150 checklists shows it as a user friendly, effective, and feasible tool.