Preparedness for a smallpox outbreak: comparing metrics for assessing levels of vaccination among health-care workers by state

Summary

By mid-2005, less than 17% of smallpox vaccine doses distributed to American states for health-care workers (HCWs) during the CDC campaign had been used. To understand how states responded, vaccination patterns were studied. Metrics were calculated to compare the level of preparedness for a smallpox outbreak in terms of absolute numbers of HCWs vaccinated compared to the percentage of doses distributed to each state, the rate of vaccination per capita population, and the percentage of HCWs vaccinated compared to the number the CDC recommended. States were then ranked. Results showed that rankings for all four metrics were statistically different (P<0·0001). In addition, when ranks were assigned to quartiles, the states directly affected on 9/11/01 ranked lowest and states widely perceived to be at lower terror risk ranked in the top. These results underscore the need to critically examine how to define an appropriate level of preparedness for a smallpox outbreak.

INTRODUCTION

The threat of bioterrorism includes concern over the potential weaponization of smallpox. In response to this possible threat, the Centers for Disease Control and Prevention Advisory Committee on Immunization Practices recommended that smallpox vaccination be administered to selected civilian volunteers [1, 2]. In January 2003, the Smallpox Pre-Event Vaccination programme for volunteer hospital-based and public health workers in the United States was initiated, with an initial goal of having 500 000 workers vaccinated [3, 4]. When undertaken, it was described as the first phase of a larger campaign to vaccinate the entire US civilian population [2]. This has never occurred, however, and the programme has not been further promoted beyond its initial phase. The Institute of Medicine (IOM), Committee on Smallpox Vaccination Program Implementation, has documented the scientific and policy challenges and controversies surrounding the initiation and implementation of the smallpox vaccination programme [5].

The Phase I plan recommended that every US hospital vaccinate 50–100 health-care workers (HCWs) to form a Smallpox Health Care Team [5]. These teams would include physicians, nurses, mid-level practitioners and ancillary staff such as housekeepers, security staff, and administrators [6]. The CDC asked all states and the District of Columbia to issue a pre-event smallpox vaccination plan by 9 December 2002 and estimate the number of vaccine doses needed [7]; it also provided states with a series of smallpox training opportunities to increase the knowledge and skills of public health and HCWs on vaccine distribution and smallpox diagnosis [8]. In addition, the CDC developed educational materials, provided technical assistance, held regular conference calls with state public health officials, and worked on overall communication plans [5].

While it was anticipated that this phase would be implemented rapidly (over 30 days), in fact, only 39 579 individuals, or <17% of the distributed doses, have voluntarily accepted vaccination since the programme's inception [9, 10]. Many issues impeded the success of the programme; logistical and economic issues hampered vaccination in some states. In other states, lengthy training sessions were required for all potential vaccines or prohibitive health inclusion criteria were mandated limiting the number of potential vaccinees. The General Accounting Office [11], in a report on the progress of the campaign, noted that states reported that they lacked guidance about what ‘smallpox preparedness’ meant and about how to assess if they were sufficiently prepared. A confluence of events affected the implementation of the campaign. First, there were a number of negative press reports on reported cardiac complications due to the vaccination of military personnel, with experts postulating that giving the vaccine to an adult population was inherently dangerous because of the presence of more contraindicated conditions [12]. In addition, the decision to use existing live vaccinia vaccine, rather than wait until the development of a killed vaccinia vaccine which would have eliminated a number of the negative health events associated with smallpox vaccination, was discussed frequently in the media and by professional medical organizations [13] and advocacy organizations [14–16]. This negative communication climate, and lack of support by leading institutions such as the American Nurses Association and a number of reputable hospitals, reduced the willingness of HCWs to be vaccinated in what most considered a rushed and careless manner.

Studies show that HCWs acceptance and adoption of the programme was very low. In fact, a number of studies looking at attitudes of HCWs towards the programme revealed that most did not think the benefits of the vaccine outweighed the risks in an adult population, which might have a number of contraindicated conditions [17–22]. For example, Yih et al. [22] studied 1165 emergency-room (ER) or intensive care unit (ICU) HCWs and found that only 32% of respondents would report to work after a patient with smallpox was admitted to their facility, if the respondent had not been vaccinated recently. Only 61%, however, reported they were willing to be vaccinated at the time of the survey. While other studies have found similar rates of intended vaccination [18–21], actual vaccination rates are much lower. Benin et al. [17] found that only two of 141 surveyed physicians actually received vaccination when asked to do so.

Because of the reticence of HCWs to be vaccinated, the CDC has not attempted to vaccinate the public against smallpox. Nor to date has CDC issued a metric for assessing the numbers of HCWs who should be vaccinated in a state to provide a reasonable level of preparedness against a perpetrated smallpox outbreak.

Purpose

The purpose of this study was to compare metrics for assessing the level of preparedness to handle a smallpox outbreak based on the level of vaccination of HCWs in each state. We constructed metrics to approximate levels of preparedness, and used these to rank states to develop a ‘preparedness’ rubric relative to the numbers of HCWs vaccinated. The metrics constructed were: (1) vaccines actually administered as a percentage of doses requested by each state; (2) numbers of HCWs vaccinated per capita population; and (3) HCWs vaccinated as a percentage of the estimated number the state would have needed to vaccinate to meet the CDC recommendation of vaccinating 50–100 HCWs for each hospital in a state. An additional goal was to use these rankings to develop quartiles of ‘preparedness’ to understand how states differed in the ‘success’ of their smallpox vaccination programme. This approach moves beyond reporting the absolute numbers of persons vaccinated by state to rates that allow comparison across states of different size. It specifically attempts to address the critical issues of the size of the vaccinated health-care workforce relative to the total population (vaccinations per capita population), the ability of states to vaccinate the numbers they anticipated needing vaccination, and the numbers actually vaccinated relative to the CDC recommendations.

METHODS

Data were obtained on smallpox vaccination for each of the 50 states and the District of Columbia from a number of sources. Number of doses of vaccine requested by the states and distributed and the number of individuals vaccinated as of 6/30/05 were obtained from CDC records [9, 10]. State population figures were obtained from the US Census 2003 estimated population [23]. Rates of vaccination for ER personnel were calculated by identifying the number of hospitals in each state [24] (obtained from: www.hospitallink.com) and multiplying the number by 75 to approximate the number of HCWs needing vaccination if all hospitals in the state met the CDC recommendation of vaccinating 50–100 persons per hospital. This method, when summed, yielded an estimated 450 375HCWs to be vaccinated in the United States. This figure is consistent with the CDC's estimate calling for a total of 440 000–500 000 HCWs to be vaccinated [2].

Although the CDC shipped vaccine directly to three cities (Los Angeles, New York City, Chicago), states and two territories (Puerto Rico and Palau), numbers vaccinated in all programmes were aggregated to calculate the number of HCWs vaccinated by state. Data for territories are not reported here.

Standardizing the absolute numbers vaccinated into three different types of vaccination rates allows meaningful comparisons of the levels of smallpox vaccination among the states. Each also provides a slightly different perspective on level of smallpox vaccination preparedness by state.

RESULTS

Table 1 includes all state-level data and metrics calculated. Column 1 is state population. Column 2 shows the estimated recommended number of hospital personnel to be vaccinated to meet the CDC recommendations of 50–100 per hospital. Column 3 is the number of doses of smallpox vaccine requested by each state and distributed by the CDC as of 30 June 2005. The last four columns, described below, are absolute number vaccinated as of 30 June 2005 (column 4); the percentage of doses requested that were actually used (column 5); the per capita vaccination rate (column 6); and the percentage of recommended number of hospital personnel actually vaccinated (column 7). Columns 4–7 also show how each state ranked against all other states on that metric.

Table 1. Smallpox vaccination metrics calculated as (a) percentage of distributed doses actually used, (b) vaccinations per capita population and (c) percentage of recommended number vaccinated for all hospitals in the state*

* Number of hospitals in each state (www.hospitallink.com) multiplied by 75.

Column (4) compared to column (5) rankings: χ²=68·64, P=0·0412.

Column (6) compared to column (7) rankings: χ²=88·76, P=0·0006.

Columns (4), (5), (6) and (7) compared: χ²=122·36, P<0·0001.

Rankings for absolute numbers vaccinated

In absolute numbers (Table 2), the five states that vaccinated the most HCWs were Texas (n=4632), Florida (n=4041), Tennessee (n=2429), Ohio (n=1921) and California (n=1854). In absolute numbers, the five states that vaccinated the fewest HCWs were Nevada (n=17), Rhode Island (n=36), Arizona (n=39), Maine (n=63) and Alaska (n=96). The states with the lowest absolute numbers of personnel vaccinated are some of the least populous states, making absolute numbers difficult to compare. When the data are standardized into metrics shown in columns 5–7 of Table 1, more meaningful comparisons are possible. For example, while Florida and Texas vaccinated the largest absolute numbers, neither state ranked in the top five based on percentage of vaccine doses actually used (Table 2), or per capita vaccination (Table 3). Florida was, however, in the top five when ranked by the percentage of recommended number vaccinated per hospital when aggregated for the state as a whole (Table 3).

Table 2. States with the highest and lowest level of preparedness calculated as absolute number vaccinated and percentage of distributed vaccine doses actually used

Table 3. States with the highest and lowest levels of preparedness calculated as vaccination rates per capita population and by percentage of CDC recommended numberof hospital personnel to be vaccinated

Vaccination rates and rankings as a percentage of doses distributed

As shown in Table 2 (excerpted from column 5 in Table 1), states varied by the percentage of doses of vaccine distributed than they actually used. Comparing the top five ranked states, Oklahoma used 53·7% of the doses received (376/700 doses) followed by Iowa (49·2%, 492/1000 doses), Nebraska (36·8%, 1470/4000 doses), Minnesota (32·8%, 1476/4500 doses) and Alaska (32%, 96/300 doses). In the five states ranked lowest, the percentage of vaccine used was only 1·1% in Nevada, 2·1% in the District of Columbia, 2·1% in Maine, 2·6% in Illinois and 3% in Rhode Island.

Vaccination rates and rankings per capita population

There is also wide variation in vaccination rates per 100 000 population (Table 1, column 6). These ranged from a high of 96·4/100 000 in South Dakota to only 0·7/100 000 population in Arizona. Table 3 summarizes smallpox vaccination rates per 100 000 population for the states with the highest and lowest rates. The four states with the highest rates are in the Midwest: top-ranked South Dakota vaccinated 96·4/100 000; Nebraska and Wyoming vaccinated 84·5 and 82·6/100 000 respectively. North Dakota and Arkansas vaccinated 65·5 and 41·8/100 000 respectively. The lowest per capita vaccination was in Arizona, with only 0·7/100 000. The other states with the lowest per capita vaccination rates were Nevada (0·8/100 000), Georgia (2/100 000), Pennsylvania (2·5/100 000) and Massachusetts (2·9/100 000).

Vaccination rates and rankings relative to CDC recommendations per hospital

Using the estimated number of hospital personnel vaccinated as a percentage of the estimated number recommended by CDC for each state, the states achieved rates ranging from 23·5% in Connecticut to only 0·6% in Arizona (Table 1, column 7). Table 3 illustrates the smallpox vaccination rate as a percentage of the estimated number of health-care personnel that would have had to have been vaccinated in each state to meet the CDC recommendation of vaccinating 50–100 per hospital. Connecticut and Tennessee had the highest rates, each vaccinating 23·5% of the recommended number of HCWs per hospital. Florida ranked third, vaccinating 22·1% of their personnel. Nebraska vaccinated 20·9% and Wyoming vaccinated 18·4%. Among the states with the lowest rates in addition to Arizona, were Nevada (0·7%), Georgia (1·2%), Rhode Island (1·6%) and Maine and Massachusetts (each 1·9%).

Doses requested by states compared to CDC recommendations per hospital

When comparing the number of doses requested by each state to the estimated number needed to vaccinate at the CDC recommended level per hospital (Table 1, columns 1 and 3), there is also wide variation in the number of doses requested by the states. Some states (n=12) requested doses that fall within the range recommended by the CDC, based on the number of hospitals in the state. The majority of states, however, requested too many doses of vaccine (n=7) or too few doses (n=32) based on the number of hospitals in the state. To vaccinate 50–100 HCWs per hospital (the CDC target) California, for example, the most populous state with the largest number of hospitals, would have needed approximately 35 000 doses of vaccine, but only requested 19 300. Similarly, Minnesota, with a large number of hospitals, would have needed approximately 20 000 doses but only requested 4500. Wisconsin would have needed 10 000 doses but only requested 2500. In contrast, Florida, which has a large population but fewer hospitals per capita, requested 24 000 doses of vaccine when they would have only needed approximately 18 000 to meet the recommended target. The District of Columbia, which has few hospitals within the district limits, was estimated to need 1300 doses to meet the target but requested 5000.

Smallpox vaccination rankings by quartile

Table 4 summarizes how the states ranked by quartiles of preparedness for each metric. This analysis illustrates how little consistency was found in how states ranked. For example, some states fell into the first quartile in absolute number vaccinated but the third or fourth quartile of states on the other metrics. New York, in the first quartile on absolute numbers vaccinated, fell in the third quartile for percentage of doses used, the fourth quartile for vaccination rate per 100 000 population and the third quartile for percentage of recommended HCWs vaccinated. On the other hand, some states that ranked in the bottom quartiles on absolute number vaccinated ranked in the top quartiles on the other metrics. For example, Wyoming ranked in the third quartile for absolute number vaccinated but the second quartile on percentage of doses used, and the first quartile on per capita population and percentage of recommended HCWs vaccinated. Similarly, Vermont ranked in the bottom quartile for both absolute number and percentage of doses used, but the second quartile in per capita population and recommended HCWs vaccinated.

Table 4. States' rankings in quartiles by preparedness metric

Statistical analysis of rankings of levels of preparedness

To assess the comparability of the metrics calculated for gauging preparedness for a smallpox outbreak, we used the non-parametric Friedman test to assess the null hypothesis that there were no differences in the distributions of the ranks (assigned to each of the metrics calculated for all 50 states and the District of Columbia). The probability distribution of Q under the null hypothesis was approximated using a χ² distribution with 50 d.f. Comparing the ranks of each state on absolute numbers vaccinated with the percent of doses distributed that were actually used, an indicator of state level response to the CDC vaccination mandate, there is a statistically significant difference (χ²=68·64, P=0·0412). Comparing the rate vaccinated per 100 000 population with the percentage of CDC-recommended hospital personnel to be vaccinated who were actually vaccinated for all hospitals in the state (χ²=88·76, P=0·0006), two standardized indicators of preparedness, is also statistically significantly different. In addition, comparing the rankings for all four metrics (two indicators of state-level response and two standardized indicators of preparedness), the results are statistically significant (χ²=122·36, P<0·0001).

Statistical analysis was also done to assess the quartile ranking for each state for the four metrics. Comparing the quartile rankings of each state on absolute numbers vaccinated with the percent of doses distributed that were actually used, there is a statistically significant difference (χ²=67·31, P=0·0517). Comparing the quartile rankings for the rate vaccinated per 100 000 population with the percentage of CDC-recommended hospital personnel to be vaccinated who were actually vaccinated, is also statistically significantly different (χ²=87·43, P=0·0008).

Finally, comparing the quartile rankings on all four metrics, the results are also statistically significant (χ²=117·3, P<0·0001). In sum, each of the four metrics, as well as the quartile rankings, that might be used to assess state level of preparedness is statistically different from the others. Thus the states varied in terms of how they ranked for preparedness depending on the metric selected.

DISCUSSION

Metrics that might be used to assess the level of preparedness for having an adequate number of HCWs vaccinated in the event of a smallpox outbreak reflect the lack of clarity and absence of consistency in implementation of smallpox vaccination in the states. The absolute number of smallpox vaccinations, as well as the three vaccination metrics calculated, underscores the low level of acceptance of the vaccination programme by both the states and HCWs themselves. The great variability in the rankings of states using each metric also shows that there are important regional differences in response to the CDC's recommendations for pre-event vaccination. We believe that it is important to study the political, socio-economic and cultural factors that influence vaccination decision-making if we are to understand the response to this vaccination campaign. Research is also needed on the types of communication strategies used by the states to persuade HCWs to be vaccinated.

The metrics calculated also highlight the importance of critically examining the way in which the ‘success’ of a vaccination campaign is measured. Because the states had wide latitude in deciding how to define their own level of preparedness [5] in terms of numbers they needed to vaccinate, it is useful to examine how the percentage of doses of vaccine the states requested and received was actually used. In addition, states may have had different ideas on the number of hospitals in their states that would need to participate in the vaccination programme, thus, the differences we found in the number of estimated HCWs for each state compared to the number of doses of vaccine actually requested. We emphasize that federal estimates of state needs may not have been consistent with what the states felt they needed. In the view of some public health officials, the states are inherently better at estimating their capacities. Thus the metrics used in this analysis were an attempt to standardize the numbers of HCWs so they could more easily be compared. Using these metrics, if we consider only the percentage of doses requested vs. actually used, states that requested less vaccine than needed to meet the target level of vaccination recommended by the CDC could claim to have been more ‘successful’ in implementing their vaccination campaign than states requesting more vaccine but actually using a smaller percentage.

For example, Oklahoma vaccinated 376 people with the 700 doses requested and received, using 53·7% of the doses requested. When looking at the number of hospitals in the state, however, Oklahoma would have needed an estimated 9675 doses to meet the CDC recommendation of vaccinating 50–100 HCWs per hospital. From this perspective, Oklahoma only vaccinated 3·9% of the number of HCWs recommended by the CDC. Another disparity uncovered by this analysis is the unique differences in rankings when looking beyond the absolute number vaccinated. For instance, although Texas and Florida had the highest absolute numbers vaccinated, they did not vaccinate at the highest rates when calculated as a percentage of doses requested that were actually used, per capita vaccination, or the percentage of the CDC target that were actually vaccinated. Thus Texas, which had the highest absolute number vaccinated (4632), ranked only 23rd on the percentage of requested doses actually used, 15th on per capita population, and 10th on the percentage of the CDC target actually vaccinated. Florida, which vaccinated the second highest absolute number of people (4041), ranked 20th on the percentage of requested doses used, 12th in per capita vaccination, and 3rd in percentage of the CDC target actually vaccinated. From a preparedness perspective, there is lack of clarity as to which metric is most appropriate as an indicator of preparedness for a particular state.

It is also interesting to note that the three states directly affected by the terrorist attacks of 9/11 (New York, Pennsylvania and Virginia) rank in the bottom quartiles among all states for smallpox vaccination preparedness for most metrics calculated. New York ranked 37th in percentage of doses actually used, 40th in per capita vaccination and 33rd in the percentage of CDC-recommended vaccinations. Comparing these metrics respectively for the other affected states, Pennsylvania ranked 46th, 48th and 45th (vaccinating 308) and Virginia ranked 39th, 32nd and 19th (vaccinating 914). New Jersey, which suffered significant casualties during 9/11 and vaccinated 671 ranked 36th in the percentage of doses actually used, 39th in per capita vaccination, and 23rd in the percentage of CDC recommendations. In contrast, the states widely believed to be at less risk of a terrorist attack, e.g. those in the Midwest with low population density, consistently rank in the top quartile of preparedness for all of the metrics calculated. For example, Nebraska had the best overall rankings for the metrics used in this analysis, vaccinating 1470 individuals and ranking 3rd in percentage of doses distributed, 2nd in per capita population and 4th in number of estimated HCWs. We do not have data to explain this phenomenon, but believe that understanding why some states (e.g. Nebraska, North Dakota, South Dakota, Wyoming) were more successful than others in vaccinating the CDC-recommended numbers of HCWs warrants further study.

CONCLUSION

Overall, the levels of smallpox vaccination in the states raise many questions about the development and implementation of a national plan for vaccination of emergency response teams to handle an infectious disease outbreak. While smallpox has received considerable attention, the likelihood of outbreaks of other infectious diseases including SARS and Avian flu underscore the need to understand state-level response to CDC recommendations for handling an outbreak before it has spread widely. The IOM notes that the CDC's rapid implementation of the smallpox vaccination programme did not allow time to finalize or test many components of the campaign. The rate of vaccination rose gradually after the campaign was launched in January 2003 but dropped precipitously by the summer of 2004. While many explanations have been given for the failure of the campaign to reach target levels, including the fact that it was launched without evidence of an impending crisis, the issue of state-level response needs greater attention. For instance, creation of a more thorough plan on what the correct ‘mix’ of HCWs in hospitals needing vaccination is important (i.e. physicians, nurses, PAs, support staff, etc.). Having a number of different hospital workers vaccinated in hospitals would allow for hospitals to institute stand-alone smallpox isolation wards that could fully function because all workers were vaccinated and there would be no time lag in waiting for newly vaccinated workers to develop immunity. In addition, there has been little done to address vaccination in non-hospital workers. Although the second phase of the CDC plan was to vaccinate up to 10 million first responders and public health workers, this was not implemented nor has a subsequent plan been put forth on what level of vaccination should be accomplished in these groups to constitute state preparedness. It should be noted that hospital smallpox readiness is but one piece of the preparedness puzzle; it is clear that the failure of the first phase of the plan has created a ‘holding pattern’ in developing further plans for being prepared if a smallpox bioterror event should occur.

In addition, investigation into why there was great variability in state vs. federal estimates of needed vaccine is required. As this analysis illustrated, some states requested much more or much less vaccine than would have been anticipated given the number of hospitals in the state. This paradox is not easily explained and further research is needed to understand this phenomenon. One strategy may be to study how states interact with federal officials, particularly the CDC, to implement vaccination programmes in the face of information that is always incomplete, and in recognition of the fact that the capacity for rapid response is a key element in preparedness under all circumstances. It may be that the uncertainty linked to the smallpox vaccination campaign, and the emotionally charged atmosphere in which it developed, created an unusual circumstance in how states responded to federal recommendations. Comparing this experience to other federal public health mandates for states would be useful in understanding why the smallpox vaccination policy failed and how to increase compliance in the future.