The Public Health Emergency Preparedness (PHEP) program is administered by the Centers for Disease Control and Prevention (CDC) to assist state, tribal, local, and territorial (STLT) public health departments develop and strengthen the partnerships, infrastructure, and capabilities needed to prepare for, respond to, and recover from public health emergencies. Since 2005, Reference Martinez, Talbert and Romero-Steiner1 the PHEP Cooperative Agreement provides support for public health emergency response infrastructure and personnel. In support of the PHEP, the CDC’s Division of State and Local Readiness (DSLR) introduced 15 capabilities or national standards to monitor progress toward public health preparedness and response. 2 The PHEP cooperative agreement provides a critical source of funding needed to sustain the ever-changing capability and capacity needs of the STLT for public health emergency preparedness and response. With the increased frequency of public health emergencies such as infectious disease outbreaks and natural disasters, planning and responding to emergencies requires continual coordination and collaboration between government and nongovernmental agencies to reach all sectors of the community. 3,4 After-action reports and improvement plans (AARs/IPs) are retrospective analyses that capture key information and best practices from emergency response exercises and real incidents. The information can be used for future improvement and corrective action. AARs provide a documented data source to evaluate the current capacity and capabilities at federal, tribal, state, local, and territorial level and are created by public health departments. Continuous quality improvement (CQI) is facilitated when a systematic approach is applied to how AARs are reported and evaluated between exercises and real incidents. Reference Singleton, DeBastiani and Rose5,Reference Donohue and Tuohy6
The utility or value of AARs has been questioned. 7 For example, most public health departments conduct and complete AARs, but there is limited analysis of the methods used or consistency applied across the AARs by the states. Standardized parameters and metrics have not been placed into practice. Therefore, AARs are usually very heterogeneous in format and report content. In our analysis, we evaluated AARs of exercises and real-world incidents to identify key factors external to the program that could influence the success of a jurisdiction’s ability to respond. In this analysis, we identified 6 reoccurring themes that were found in most of the AARs: Communications, Coordination, Resource Distribution, Unified Planning, Surveillance, and Knowledge. Although frequency does not indicate correlation or causation, it is an indication of the priority issue a state is investing resources for a particular response.
The 15 PHEP capabilities create a useful framework to support a more standardized analysis of exercises and real-world incidents. PHEP Cooperative Agreement awardees can structure their AARs for exercises, natural disasters, and incidents involving Chemical, Biological, Radiological and Nuclear (CBRN) threats around the PHEP capability standards and submit their AAR findings to CDC. CDC PHEP cooperative agreement awardees typically develop their AARs based on the revised 2018 capabilities. Awardees submit AARs to CDC for review following full scale exercises and major incidents that impact public health such as infectious disease outbreaks, natural disasters, and CBRN events.
The development of the Homeland Security Exercise and Evaluation Program (HSEEP) in 2008 by the Federal Emergency and Management Agency (FEMA) provided a structured method for consistent planning and evaluation of emergency exercises. 8 Similarly, evaluations have been conducted on the quality of AAR data used for CQI from public health systems and have determined the need for a systematic way of capturing strengths, weaknesses, and areas of improvement to establish common evaluation metrics. Reference Savoia, Agboola and Biddinger9
We evaluated 2 types of AARs: exercises and real-world incidents. Exercises were focused on responses requiring the use of medical countermeasures. Real-world incidents were focused primarily on natural disasters and infectious disease outbreaks. Inductive qualitative analyses from AARs generated between 2012 and 2018 were conducted using ATLAS.Ti software, version, 8.1, Cleverbridge, AG. Fourteen AARs that had been submitted to CDC/DSLR were chosen for qualitative analysis and were used to create the coding scheme that is presented here. AARs were chosen from various public health events and exercises. AARs events included: 8 natural disasters (hurricanes, mud slides, etc.), 2 infectious disease outbreaks (Hepatitis A and 2014 Ebola), and 4 full-scale exercises (Pills to Polar Bears [anthrax], Big Sky Push [Pandemic Influenza], Vigilant Guard [plague and earthquake], 2019 Health and Human Services [HHS] Crimson Contagion Functional Exercise). Reference Lowe, Salame-Alfie and Neurath10,11 Inductive coding was used to identify cross-cutting themes across all events and the 15 PHEP capabilities. The co-authors divided into 2 teams of 2 reviewed and coded each AAR. Passages were labeled and categorized under themes and sub codes in each AAR using ATLAS.Ti. Frequency of themes and sub codes were calculated as the total number identified among a group of statements reviewed across AARs. A standardized analysis format is proposed for future use in the evaluation of AARs.
Discussion
While themes and subthemes vary depending on the type of AAR, 6 novel overarching themes that emerged as a result of this coding analysis. The themes identified were: Communications (N = 281), Coordination (N = 270), Resource Distribution (N = 108), Unified Planning (N = 129), Surveillance (N = 21), and Knowledge Sharing (N = 27) (Table 1). Communications is the ability to warn and share information with the public and incident management personnel. During responses and exercises, communication with external partners to health departments was the most identified subtheme, which highlights the importance of communications during public health emergencies. These partners included other state and federal government agencies, first responders, and nongovernment organizations. Situational awareness and information sharing accounted for most of the different types of communications to partners and stakeholders (N = 54; 73%). According to these subthemes, internal communication of information to improve relevant operations and systems ranks high in importance during responses involving an incident management system. Written communication (N = 31; 33%) and oral communication (N = 23; 24%) were the most common methods of communication discussed or identified during a response. Written communication methods included newspaper, fact sheets, and reports. Oral methods included in-person meetings and telephone calls.
Table 1. State or local health department activities identified in AARs by themes and subthemes
Abbreviations: WebEOC=Emergency Management Software; ICS/EOC=Incident Management Structure/Emergency Operations Center; POD=Points of Dispensing; PPE=Personal Protective Equipment; POD/RSS=Point of Dispensing/Receipt, Stage, and Store.
Coordination is the ability for jurisdictions to understand other sectors’ roles and responsibilities and mount a multisector response. Coordination specifically with partners (N = 102; 31%) was the most common sub-code, and states considered it a critical element when responding to an emergency. Coordination for staffing (N = 78; 24%) a state response was the second most frequently mentioned sub-theme. Resource Distribution is the process by which jurisdictions obtain and distribute needed resources. There were 3 critical elements to resource distribution that were considered equally important. These included (1) management of assets, (2) distribution of assets, and (3) points of distribution: receipt, stage, and storage sites. Unified Planning is the ability to ensure planning and training involves coordination between individuals and organizations and formalizes documents that assign roles and responsibilities during an emergency. Surveillance uses information systems to monitor health data and provide analysis and interpretation to provide early warning of health threats. There were fewer surveillance codes (N = 21; 2%) in comparison to other coded themes. This could indicate that surveillance is a well-established response capability that needs fewer corrective actions. Knowledge Sharing (N = 27; 3%) which is the exchange of health-related information and situational awareness data among all levels of government and private sector, was also mentioned less frequently in the AARs. It was not statistically significant to include data on Surveillance and Knowledge Sharing in Table 1.
Limitations
The assessment is based on 14 AARs, and most of the exercises reviewed only included a singular capability (medical countermeasures). Historically, the majority of exercises in the PHEP program have a distribution and dispensing focus. Therefore, the number of exercises of other capabilities was not readily available for analysis and there may be additional themes not captured in this study. The exercises and events were randomly chosen to provide a cross-representation of the different types of AARs. The 6 themes account for cross-cutting activities for evaluation of AARs and are not a replacement for or recommended over the capabilities, which are intended for PHEP evaluation.
Conclusions
Every AAR reviewed in this study included the 6 themes identified (Communications, Coordination, Resource Distribution, Unified Planning, Surveillance, and Knowledge Sharing). These themes support public health system capabilities. We propose that these 6 themes be used across all capabilities to offer a cross-cutting framework that is not restricted to the capabilities. The themes are a replacement for the capabilities in the AARs. ARRs written after Full Scale Exercises followed Homeland Security Exercise and Evaluation Program (HSEEP) guiding principles structured by capability. (https://emergency.cdc.gov/training/ERHMScourse/pdf/127961885-Hseep-AAR-IP-Template-2007.pdf). A more focused approach on writing and reviewing AARs by the constructs rather than capabilities may allow practitioners to capture more clearly defined strengths and weaknesses across multiple types of exercises and incidents. Additional focus should be given to the inconsistent reporting because incidents were not always reported by capability, but by strengths and weaknesses with an improvement plan. Incidents were not in a consistent format, which also made it difficult to evaluate. A standardized format for reporting, including Web-based, for the purposes of analysis would ensure consistent coding for both incidents and exercises and reduce the workload of public health staff.
Evaluating AARs presents an opportunity for additional focus on improving operational and system-wide readiness and performance for public health preparedness with corresponding action plans for areas that need improvement and accountability in program operations and response. Use of a standardized analysis format of AARs is recommended.
Acknowledgments
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
Author contribution
The manuscript was jointly written and edited by the authors. Each author participated in analyzing at least 1 AAR.
