Introduction
In this article, we present the results of a randomized controlled trial testing whether a postcard reminder increased senior citizens’ uptake of recommended vaccines. Our intervention is oriented toward directly changing patient behavior (rather than attitudes). In other words, it falls in the category of behavior-oriented interventions that are noted as promising in systematic reviews such as Betsch et al. (Reference Betsch, Bohm and Chapman2015) and Brewer et al. (Reference Brewer, Chapman, Rothman, Leask and Kempe2017). Behavior-oriented interventions target one subset of the broader determinants of vaccine uptake; these broader determinants include vaccine confidence, constraints and calculation (for a review, see Betsch et al., Reference Betsch, Schmid, Heinemeier, Korn, Holtmann and Böhm2018). Prior research on behavior-oriented interventions has indicated that reminders in particular can be effective tools for changing health-related behaviors, including vaccine uptake (Stone et al., Reference Stone, Morton, Hulscher, Maglione, Roth, Grimshaw, Mittman, Rubenstein, Rubenstein and Shekelle2002), though reminders are not uniformly successful (Bourgeois et al., Reference Bourgeois, Simons, Olson, Brownstein and Mandl2008). Given hitherto promising but mixed results in this category of interventions, understanding the conditions under which behaviorally informed interventions such as mailed reminders are most effective is an ongoing area of research (Brewer et al., Reference Brewer, Chapman, Rothman, Leask and Kempe2017).
This study makes three primary contributions. First, and most importantly, it contributes to the literature on vaccine uptake by analyzing the efficacy of a postcard reminder sent to senior citizens in southern USA during the 2017–2018 influenza season. Second, we contribute to the behavioral insights literature by presenting the results of a study in which the intervention had several behaviorally informed features, such as simple messaging and a call to immediate action, and by identifying contexts that may increase the probability of success for these types of interventions. Finally, we carry out our study using methods that allow causal identification even in situations where a ‘pure’ control group is not feasible. As such, our methods can speak to a broad audience of researchers operating in applied contexts who, for various reasons, are unable to withhold treatment from any study participants, but who still wish to rigorously estimate the causal effect of their intervention.
The study was carried out during the 2017–2018 influenza season in Louisiana through a collaboration between the Office of Evaluation SciencesFootnote 1 at the US General Services Administration and the Louisiana Department of Health. We evaluated the effectiveness of a vaccination reminder postcard sent to Louisiana residents aged 65–70. The postcard referenced four vaccines typically recommended for people in this age group: influenza, shingles, tetanus and pneumonia. This postcard was sent during the 2017–2018 fall–winter season to residents who were: (1) in the target age bracket; and (2) listed as (over)due at least one of the four vaccines in the Louisiana Immunization Information System (IIS). Due to the preference not to disrupt an existing reminder program, we could not withhold the postcard from a control group of Louisianans. Additionally, logistical constraints meant that the postcards could not be sent out all at once, but rather needed to be staggered over a period of four months. We used the staggered nature of the mailings to randomly assign when eligible Louisianans received the postcard. The result was a stepped wedge research design in which we could identify the causal effect of the postcard on the probability of receiving any of the four vaccinations during the study period.
We find that the postcard positively affected the average number of four recommended vaccines received by study participants. The treatment effect is substantively largest and statistically detectable among individuals who receive the postcard in October. In addition to these main findings, we explore the effect of the postcard reminder on each of the four types of vaccines and find that the postcard boosts uptake of the flu and shingles (but not pneumonia or tetanus) vaccines. We also explore whether sending the postcard reminder in October is effective because individuals who receive the postcard in October have the postcard for more days compared to the other months or because the month of October is different. We find that the month of October may be particularly important for vaccination reminders. We discuss the implications of our findings for practitioners and use exploratory analyses to develop hypotheses for future research.
Encouraging vaccination
We study an intervention that was designed to increase compliance with the Centers for Disease Control and Prevention's (CDC) recommended schedule of vaccinations for 65–70-year-old US residents. The CDC recommends that people in this age range should receive vaccinations against four illnesses: influenza, tetanus (Td/Tdap), pneumococcal pneumonia and herpes zoster (shingles) (CDC, 2018a). Actual vaccination rates for these four vaccines currently do not meet official immunization rate targets, as identified by the CDC as part of the Healthy People 2020 objectives. Specifically, Healthy People 2020 identified the following vaccine-specific goals: that 30% of adults over 60 years of age receive a zoster (shingles) vaccine, 90% of adults over 65 years of age receive a pneumococcal vaccine and 90% of adults over 65 years of age receive a seasonal influenza vaccine (US Department of Health and Human Services, Office of Disease Prevention and Health Promotion, 2018). Rates of compliance with these recommendations are generally under target, but vary significantly by geographic and demographic indicators. It is in this context of imperfect compliance with recommended vaccine schedules that we evaluate the effectiveness of a postcard reminder to receive one's recommended vaccinations. In this section, we review the recommendations for these vaccinations in more detail, illustrating the public health relevance of this intervention.
The influenza (flu) vaccine reduces the risk of getting the seasonal flu; the effectiveness of the vaccine varies significantly from year to year, but one study estimated average effectiveness across years to be around 59% (measured as reduction in the risk of contracting the flu) (Osterholm, Reference Osterholm, Kelley, Sommer and Belongia2012). The CDC's initial calculations of effectiveness for the 2017–2018 influenza vaccine estimate a 36% reduction in risk among the vaccinated compared to the unvaccinated, with a confidence interval of 27–44% (CDC, 2018b). This estimated effectiveness is higher than was anticipated early in the 2017–2018 flu season, which saw a more dangerous flu season than usual (McNeil, Reference McNeil2018). Prior analyses have indicated that provision and uptake of the flu vaccine is typically cost-effective (Lee, Reference Lee, Bailey, Wiringa, Afriyie, Wateska, Smith and Zimmerman2010).
Among adults in the USA, vaccination rates are particularly low for the influenza vaccine: an estimated 43% of adults and 59% of children received the vaccine during the 2016–2017 flu season (CDC, 2017). The flu becomes more serious as people age; among the elderly, the flu is associated with increased risks of complications and mortality. Flu vaccine uptake among the elderly is higher than among adults aged 18–64: an estimated 65% of adults over 65 years of age received the flu vaccine during the 2016–2017 flu season compared with an estimated 38% of adults aged 18–64 years (CDC, 2017). However, flu vaccination rates continue to lag behind the recommended levels of 90% of adults over 65 years of age and 80% of adults between 18 and 64 years of age (US Department of Health and Human Services, Office of Disease Prevention and Health Promotion, 2018).
Elderly adults (over the age of 65) are, in addition to the annual flu vaccine, recommended to receive vaccines against shingles, pneumonia and tetanus. The exact recommended sequence varies by the patient's individual vaccination history and risk factors (CDC, 2018a), making it more difficult for patients to know when they are due a given vaccine. For example, the pneumococcal vaccine can protect against diseases including pneumonia, meningitis and sepsis and comes in two forms: 13-valent pneumococcal conjugate (PCV13) and 23-valent pneumococcal polysaccharide vaccine (PPSV23). The type and number of doses of vaccines recommended varies somewhat by age, previous vaccination history and underlying medical conditions (CDC, 2015). While there was an increase in adults over 65 years of age estimated to receive the pneumococcal vaccine from 60% in 2012 to 64% in 2015, vaccination rates still lie well below recommended rates of 90% (Williams, Reference Williams, Lu and O'Halloran2017; US Department of Health and Human Services, Office of Disease Prevention and Health Promotion, 2018). Overall, given lower-than-recommended rates of uptake for these vaccines, there is an opportunity for behavioral interventions to help increase vaccination rates.
Reasons for low compliance
The reasons for lower-than-recommended vaccination rates include practical barriers, such as cost and accessibility issues, as well as attitude-based reluctance and refusals, which can arise due to a perceived lack of efficacy or fear of side effects. These variables can jointly be termed ‘vaccine hesitancy’ (MacDonald et al., Reference MacDonald2015). Psychological models of vaccine hesitancy emphasize variables such as complacency, convenience and calculation (MacDonald et al., Reference MacDonald2015; Betsch et al., Reference Betsch, Bohm and Chapman2015). A recent contribution further suggests the addition of constraints and collective responsibility for a ‘5C’ psychological model of vaccine acceptance (Betsch et al., Reference Betsch, Schmid, Heinemeier, Korn, Holtmann and Böhm2018). These frameworks can be used to diagnose barriers to vaccine uptake and to tailor policy solutions that address those barriers. For example, a recent review of flu vaccine uptake finds that hesitancy for this vaccine is affected by a broad range of factors, but especially confidence and complacency (Schmid et al., Reference Schmid, Rauber, Betsch, Lidolt and Denker2017).
The intervention studied in this paper was inspired by a decision-science approach; this type of intervention attempts to guide behavior “without trying to change beliefs and attitudes” (Chapman, Reference Chapman2019). As such, it falls within the category of behavioral interventions, which in turn have been noted as particularly promising in two recent systematic reviews (Betsch et al., Reference Betsch, Bohm and Chapman2015; Brewer et al., Reference Brewer, Chapman, Rothman, Leask and Kempe2017). This type of intervention specifically aims to overcome factors such as convenience or complacency, while not directly addressing confidence. Even though this category of vaccine uptake interventions typically does not reduce practical barriers such as cost or distance, it can be effective at overcoming psychological barriers such as forgetfulness. Additionally, behavioral interventions tend to be relatively low cost (as is the intervention studied here), which also speaks in their favor. We thus acknowledge the position of this specific intervention type as promising, yet limited in the type of barrier it addresses. We suggest an intervention like the one studied here may be especially interesting for practitioners as a potential complement to other interventions, as combined strategies to encourage vaccine uptake may be more effective than isolated approaches (Dini et al., Reference Dini, Toletone, Sticchi, Orsi, Bragazzi and Durando2018).
Identifying and addressing barriers to vaccine uptake in Louisiana
The Louisiana Department of Health regularly runs initiatives to increase vaccine uptake in the state. In the 2016–2017 flu season (the year immediately prior to the intervention evaluation presented here), an estimated 58% of Louisiana adults aged 65 and over received the flu vaccine (the Louisiana State's Healthy People 2020 target is 70% and the national target is 90%). A total of 73.1% of Louisiana adults aged 65 and over had received at least one of two pneumococcal vaccines (the national Healthy People 2020 target is 90%).Footnote 2 As a result, the Louisiana Department of Health has ongoing work to increase vaccination rates in the state.
Barriers to vaccine uptake that the Louisiana Department of Health has identified as particularly relevant to their work include: lower-than-desired public awareness about the importance and safety of vaccinations; lower-than-desired rates of strong health care practitioner recommendations; and lower-than-desired access to vaccine providers (including issues with transportation to health care providers and community pharmacies). As discussed above, a postcard intervention is part of the behavioral intervention toolkit; as such, it may be a good tool for increasing awareness (which is related to reducing complacency in the parlance of the 5C framework of vaccine hesitancy). While the postcard is not intended to reduce practical barriers such as cost of transportation, it includes a reference to medical providers’ recommendations to get vaccinated (more detail provided below) and as such may address the lower-than-desired rates of strong health care provider recommendations. With these barriers in mind, the Louisiana Department of Health initially fielded a postcard intervention in the 2016–2017 flu season and repeated it (with the addition of a random assignment component as described in this paper) in the 2017–2018 flu season.
The postcard was deployed during the flu season in part because, with respect to the flu vaccine in particular, there is evidence that significantly more people intend to get a flu shot than actually do each year (Galarce et al., Reference Galarce, Minsky and Viswanath2011). Areas such as this, where people hold intentions to act but there is a lack of follow-through, are classic avenues in which timely reminders can be effective at changing behavior. As an additional practical consideration, more involved interventions can be significantly more resource-intensive: while it is possible to increase flu vaccine uptake rates through financial incentives and/or mandates (Pitts et al., Reference Pitts, Maruthur, Millar, Perl and Segal2014), this avenue may not always be practically feasible or ethical (Bronchetti et al., Reference Bronchetti, Huffman and Magenheim2015). Similarly, general education campaigns can be expensive yet have mixed results, sometimes achieving substantial increases (Kimura et al., Reference Kimura, Nguyen, Higa, Hurwitz and Vugia2007), but sometimes not creating any change at all (Dey et al., Reference Dey, Halder, Collins, Benons and Woodman2001; Chamberlain et al., Reference Chamberlain, Seib, Ault, Rosenberg, Frew, Cortes, Whitney, Berkelman, Orenstein and Omer2016). These considerations further supported the choice of a behavioral intervention strategy.
Reminders as tools in health care
As described above, the postcard addressed barriers to uptake that were identified locally as prominent concerns that a postcard may partially yet feasibly address. In this section, we further situate the chosen intervention in the literature on the efficacy of reminders as tools in public health interventions.
Prior research gives us reasons to expect that reminders can increase compliance with recommended vaccine schedules. For example, many adults who get vaccinated are not highly informed about vaccines (Dubé, Reference Dubé, Laberge, Guay, Bramadat, Roy and Bettinger2013); instead of actively demanding vaccines, they get vaccinated due to provider recommendations or due to social norms (perceptions that getting vaccinated is the common thing to do). For people in this category, reminders may be an effective way to convey recommendations and social norms and to increase the salience of the need to get vaccinated. In addition, Betsch et al. (Reference Betsch, Bohm and Chapman2015) point out that since people with strong anti-vaccine beliefs are a relatively small minority, it may be most productive to focus scarce vaccine promotion resources on the larger group of complacent or forgetful individuals.
Empirical research has validated that reminder-based interventions can improve health care-related compliance, including medication adherence and health-promoting behaviors (Fry et al., Reference Fry and Neff2009; Fenerty et al., Reference Fenerty2012). Stone et al. (Reference Stone, Morton, Hulscher, Maglione, Roth, Grimshaw, Mittman, Rubenstein, Rubenstein and Shekelle2002) show that reminder or recall initiatives are effective ways to increase compliance with vaccine schedules, probably because these initiatives successfully reduce rates of forgetfulness and complacency. In the specific context of flu vaccines, McCaul et al. (Reference McCaul, Johnson and Rothman2002) find that sending reminder letters to elderly individuals who did not get a flu shot in the previous year increases their uptake of the flu vaccine by 8.6 percentage points from a baseline of 20%, while Yokum et al. (Reference Yokum, Lauffenburger, Ghazinouri and Choudhry2018) find that sending reminder letters to Medicare beneficiaries increases flu vaccine uptake by between 0.4 and 0.9 percentage points from a baseline of 25.9%. However, reminders are not always effective: Bourgeois (Reference Bourgeois, Simons, Olson, Brownstein and Mandl2008) finds that sending personalized reminders to employees with a baseline rate of flu vaccination at 19% results in a statistically insignificant increase in uptake (5 percentage points). The current study adds to this accumulating body of evidence.
Postcard design
While many variables can affect the effectiveness of reminders, one element that may be consequential is the content and design of the reminder. For example, prior research has shown that vaccine uptake is significantly affected by physician recommendations (Bratic et al., Reference Bratic, Seyferth and Bocchini2016), which was also a barrier identified by the Louisiana Department of Health as relevant in this situation. While the most effective recommendations are likely made by one's own doctor in person, it is possible that highlighting health care providers’ recommendations more generally helps increase a reminder's effectiveness. The postcard circulated in this study includes a statement that the vaccine sequence is recommended by the CDC, as well as an exhortation to talk to one's health care provider about the sequence.
Another factor that may cause people to fall behind on their vaccination schedule could be failure to follow through on intentions. One way to increase follow-through is to ask people to act immediately rather than putting off the intended action to an indeterminate future time (Milkman et al., Reference Milkman, Beshears, Choi, Laibson and Madrian2011; Hagger & Luszczynska, Reference Hagger and Luszczynska2014). The postcard circulated in this study encourages the recipient to “call your healthcare provider today”; this may help increase follow-through on intentions to get vaccinated. By referencing the CDC and encouraging recipients to act immediately by calling their health care provider, the content and design of the card uses behavioral insights to increase the effectiveness of the postcard reminder.
The setting
During the 2017–2018 flu season, for the second year in a row, the Louisiana Department of Health sent out a postcard reminder to Louisianans aged 65–70. An image of the postcard appears in Figure 1. The postcard mentions all four of the vaccines recommended by the CDC for 65–70-year-old people (influenza, pneumococcal, zoster, tetanus). The postcard was sent to individuals who showed as being (over)due at least one of these four vaccines in the Louisiana IIS (also referred to as the ‘vaccine registry’).Footnote 3 Because the sample of recipients was drawn from the IIS early in the flu season, the sample includes all 65–70-year-old Louisiana residents except those who had all of their vaccinations and had already received the 2017–2018 season's flu vaccine by 9 September 2017 (the date of finalizing the study sample).Footnote 4 The upper age limit was imposed due to the Louisiana Department of Health's interest in minimizing the number of postcards accidentally sent to deceased residents (the IIS does not receive timely updates regarding deaths). The postcard was designed by Pfizer, who also funded postcard distribution in both the 2016–2017 flu season (no evaluation of the effort occurred that season) and in the 2017–2018 flu season (under consideration here).Footnote 5
Figure 1. Postcard reminder for vaccinations, showing the inside of a folded card.
Because the postcard initiative was ongoing and had in the past been sent to all Louisianans who met the inclusion criteria mentioned above, the Louisiana Department of Health preferred not to exclude any eligible residents from receiving the postcard. Additionally, logistical constraints required that the postcards be sent in batches rather than all at one time. The staggered nature of the intervention allowed us to randomly assign the timing of postcard receipt, thus overcoming the absence of a pure control group in this study.
Experimental design
We employed a block-randomized stepped wedge design to evaluate the postcard reminder intervention. The stepped wedge design, also called a waitlist design, allows researchers to evaluate a program that requires all individuals to receive treatment (Brown & Lilford, Reference Brown and Lilford2006; Hussey & Hughes, Reference Hussey and Hughes2007; Gerber & Green, Reference Gerber and Green2012). We used this design to identify the causal effect of the treatment in the context of our experiment, where every individual must receive the postcard reminder and join the treatment group by the end of the study period.Footnote 6 We additionally used block randomization to increase the precision of our analysis. We blocked by individuals’ vaccination histories, as individuals with different vaccination histories could react differently to the postcard.
Stepped wedge design
In the stepped wedge design, individuals were randomly assigned a month in which they receive treatment; the months were October, November and December of 2017 and January of 2018. We considered individuals randomized to the January 2018 month as the control group. In all analyses below, we excluded vaccinations that were received during and after January 2018 (at which point all participants had received the treatment). In the analyses, we compared the vaccination behavior of individuals assigned to receive the postcard in October, November and December 2017 (respectively) to the vaccination behavior of individuals who were assigned to receive the postcard in January 2018. Table 1 visualizes the stepped wedge design.
Table 1. Stepped wedge design. ‘Control’ indicates the month in which a group is in the control condition. ‘Intervention’ indicates the month in which a group is in the treatment condition. n = number of individuals in each group and, as a result, the number of individuals who move from the control to treatment conditions each month. Groups are made up of each of the four blocks of vaccination ‘types’. Probability of treatment assignment is the same across blocks, though block sizes within treatment groups are different.
Block randomization
We blocked treatment assignment by vaccination history because the postcard reminder might have different effects depending on the individuals’ prior vaccination behavior. The participants entered the study with different vaccination histories. Some participants might have been up to date on all vaccinations except for the seasonal flu shot. Others might have not been up to date on their vaccinations for decades. Yet others might have been up to date on some vaccinations and behind on others. Since Louisiana only sent postcard reminders to people who were (over)due at least one vaccination, individuals who were completely up to date on vaccinations were not part of the study pool.Footnote 7
We expected that these different vaccination histories were informative about the individuals’ propensity to take up vaccinations. For example, if individuals had not been up to date on vaccinations for decades, they might have made a conscious decision not to vaccinate, which would be unlikely to change as a consequence of our intervention. On the other hand, among those individuals who were up to date on some vaccinations but not others, we might find a higher proportion of individuals who were not opposed to vaccinations, but had simply forgotten that they were due a vaccination or had not gotten around to acting on a general desire to get vaccinated. We might expect postcard reminders to work well for those individuals.
Because the recommendations for the pneumococcal vaccine depended on the individual's health history and because the vaccination itself was composed of several different shots, our data (which did not include individual health information beyond vaccination status) did not allow us to decide conclusively which individuals were overdue pneumococcal vaccines. Therefore, by necessity, we relied on prior vaccination records for shingles, tetanus and influenza to block recipients based on vaccination history.Footnote 8
Individuals in each block were randomly assigned, with equal probability, to receive the postcard in one of the four treatment months. The four blocks are defined as follows:
(1) Vaccinated: individuals who are up to date on both tetanus and shingles vaccines and also received a flu shot in the flu season immediately prior to our experiment.
(2) Non-vaccinated: individuals who are overdue on both tetanus and shingles vaccines and did not receive a flu shot in the year prior to our experiment.
(3) Under-vaccinated (all but flu): individuals who are up to date on tetanus and shingles vaccinations but did not receive a flu shot in the year prior to the experiment.
(4) Under-vaccinated (mixed): individuals who were overdue some combination of tetanus, shingles and flu vaccines.
The block sizes were different, but within each block, individuals had an equal (0.25) probability of being assigned to each of the four months. Table 2 gives further detail on the blocks and numbers of individuals within each block. In total, approximately 50,000 individuals received the postcard reminder every month.
Table 2. Blocking scheme by vaccination history. The blocks are different sizes (i.e., have different numbers of individuals). Treatment assignment is distributed evenly across each of the months within each block.
a Pneumococcal vaccination histories were not part of the blocking scheme. Vaccinations for pneumococcal are included in the analysis and results.
We hypothesize that the postcard reminder will increase vaccinations among individuals in each of the treatment groups (October, November and December) compared to control individuals. This hypothesis was pre-registered with the US General Services Administration's Office of Evaluation Sciences. In the ‘Results’ section below, we further investigate whether the postcard had a greater impact after participants had been exposed to it for a longer time period and whether its impact varied by vaccine. These analyses were not pre-registered and should therefore be interpreted as exploratory and hypothesis-generating.
Data and methods
Our data were drawn from the Louisiana IIS from the Louisiana Department of Health. Our experimental pool included 208,867 elderly individuals between the ages of 65 and 70 in Louisiana who were (over)due at least one vaccination at the time of the beginning of the experiment, 9 September 2017.Footnote 9 Table 3 presents baseline descriptive statistics about vaccinations in the experimental pool. At the start of the experiment, more individuals had received a recent seasonal flu vaccine,Footnote 10 as compared to having received the non-seasonal tetanus and shingles vaccines.
Table 3. Baseline data – vaccinations (n = 208,867). Dataset includes individuals without vaccination data. All percentages include individuals with no data.
Missing data
The Louisiana IIS records the last vaccination date of each vaccine received by each resident. Vaccinations records are compiled and submitted to the IIS by doctors and pharmacists in Louisiana. However, reporting vaccination dates for adults is voluntary for care providers. This means that data in the IIS in September 2017 were incomplete; while a number of individuals might have received their vaccinations, their health care providers might not have reported the data into the IIS. For the purposes of our analysis, the absence of a vaccination record means one of two things: that the individual did not receive the vaccine; or that the individual received the vaccine but the health care provider did not report it.
The rates at which vaccination records were absent vary significantly by vaccine:
• Flu: 28% of individuals have no record of receiving a seasonal flu vaccine (in any year).
• Pneumonia: 74% of individuals have no record of receiving this vaccine.
• Tetanus: 68% of individuals have no record of receiving this vaccine.
• Shingles: 81% of individuals have no record of receiving this vaccine.
In other words, missing data were present in both treatment and control conditions, but due to data limitations, we could not distinguish missing data from non-vaccination. We chose to err on the side of caution and treat all null vaccination records as if the individual did not receive the vaccination. This means that, in the analyses below, we estimated the average treatment effect of the postcard on receiving a vaccine that was reported to the IIS. The null outcome included both individuals who did not receive the vaccine and individuals who received the vaccine but the health care provider did not report the vaccination to the IIS. This choice means that, in the results reported below, we likely underestimated the impact of postcards on vaccine uptake.
Measurement
The outcome of interest is the proportion of the four recommended vaccinations received by the end of the observation period. For example, if an individual received two out of four vaccines (e.g., flu and shingles) in the month before receiving the intervention, the individual's baseline outcome is 0.5. If the individual then received an overdue vaccine (e.g., tetanus) after receiving the postcard, the post-treatment outcome is 0.75. We describe effects in percentage point changes throughout the rest of the paper; the percentage point estimate refers to the change in the proportion of vaccines received. In addition to this joint analysis of all four vaccines, we also carried out post-hoc analyses of the results for each vaccine separately and for each treatment month separately.
Method
We use a saturated regression (Lin, Reference Lin2013) to estimate the average treatment effect of the postcard reminder. The outcome is the proportion of vaccines out of four that an individual received between 9 September 2017 and 9 January 2018. The treatment indicator denotes whether an individual is in the October, November or December treatment groups or the January control group. We also include: (1) the number of vaccines the individual received between 8 September 2016 and 8 September 2017 as a baseline indicator; and (2) the block indicator for each individual. This regression adjusts for covariates by interacting mean-deviated covariates with the treatment indicators. Additionally, because the outcome includes multiple comparisons, significance levels per outcome are reported both with unadjusted p-values and after using the Holm–Bonferroni procedure to adjust p-values (Holm, Reference Holm1979).
Results
The postcard reminder has a small but statistically significant effect on vaccination among recipients who received the postcard in October (see Table 4).Footnote 11 These results hold with and without adjusting p-values. The first set of results in Table 4 shows the average treatment effect for individuals who received the postcard in October. We estimate that, by 9 January 2018, individuals in the October treatment group received 0.27 percentage points more vaccinations (563 additional vaccinations) when compared to individuals in the control group; a difference that is statistically significant. Individuals in the November group received 0.15 percentage points more vaccinations (313 vaccinations) than the control group, and individuals in the December group received 125 more vaccinations than the control group; however, the vaccination rates in the November and December treatment groups are not statistically significantly different from the control group.
Table 4. Effect of postcard reminder on vaccination rates among the elderly.
Observations = 208,511.
Regression included baseline outcome and block indicator.
For the pre-registered hypothesis, we find that receiving the postcard in October caused higher vaccination rates. Receiving the postcard in November or December did not have a statistically significant impact on vaccine uptake, though the point estimates were positive. Going beyond the aggregate results for the pre-registered hypothesis, we can ask more fine-grained, albeit post-hoc, questions about the impact of the reminder postcard. For example, do we see these results because the postcard was particularly effective for some subset of the four recommended vaccinations? In addition, do we see time-varying results because sending the postcard in the month of October is particularly effective or because individuals in the October treatment group were exposed to treatment (i.e., had the card in their possession) for longer? In the next section, we explore questions about the type of vaccination and how treatment month versus length of treatment exposure might affect vaccination rates. As mentioned above, we consider the studies in the next section exploratory; we did not pre-register these outcomes nor make multiple comparison adjustments to the p-values of the analyses.
Effects by vaccine type
First, we explore the possibility that the postcard is particularly effective for some subset of the four recommended vaccines. To do this, we estimate the effect of the postcard on each recommended vaccine separately and report the results in Table 5. We find that receiving the postcard in October significantly increases uptake of the flu and shingles vaccines, with the flu vaccine seeing the largest increase. The null results for the November and December treatment groups persist for all four vaccines. This suggests that the observed overall increase in vaccination rates in the October treatment group is driven by increased flu vaccinations, and to a lesser extent by increased shingles vaccinations.
Table 5. Effect of postcard reminder on vaccination rates by vaccine.
*p < 0.1, **p < 0.05, *** p < 0.01.
Observations = 208,511 per vaccine.
Regressions included baseline outcome and block indicator.
Over-time variation
Both Table 4 and Table 5 show that individuals who receive the postcard reminder in October have the highest vaccination rates when compared to the control group. However, these analyses do not allow us to distinguish whether vaccination rates are higher in October because receiving a postcard reminder during the month of October is particularly effective or because having the postcard reminder for three months instead of two months or one month is more effective.
While our study was not pre-registered to distinguish between the effect of receiving the postcard in October and the effect of having a postcard reminder for three months, we can nonetheless make use of the randomization process to look at this question through exploratory analyses. Figure 2 depicts the number of vaccinations received by study participants during the observation period. Each line represents either the control group or one of the monthly treatment groups. We see that there are overall seasonal trends in the rate of vaccination, in that individuals tend to vaccinate more in October than in November or December, regardless of their assigned treatment group. We also observe weekly trends: individuals tend to vaccinate more frequently during weekdays instead of weekends. These trends cannot speak directly to the origins of the October-specific effect, but we can use further analysis to tease apart a cumulative treatment time effect from a seasonal or October-specific effect. In particular, if the number of months that an individual has a postcard is important, then we would expect individuals who had the postcard reminder for 21 daysFootnote 12 to have similar treatment effects, regardless of the month in which they received the card. On the other hand, if the treatment effect in October is truly different from the treatment effects in November and December, then we should see a difference in the estimated treatment effect even when we restrict observation to 21 days after receiving the postcard.
Figure 2. Daily vaccinations received by study participants during the course of the study by treatment month.
Figure 3 compares the proportion of individuals who received at least one of the four vaccinations within 21 days after the postcard reminder was mailed to their treatment group against the behavior of the control group in the same time period. Note that each of these three comparisons is made against the January control group. This means that we use the January control group three times at three different time points in these comparisons.Footnote 13 If the postcard reminder is more effective due to the length of time that an individual is exposed to the postcard, then we would expect the difference between treatment and control groups in Figure 3 to be fairly similar in each of the three months. However, Figure 3 shows that the difference between treatment and control groups is largest in October, and then increasingly smaller in November and December. This difference suggests that the postcard reminder was more effective in October, and that this was not due to an individual's exposure to the postcard for three months, but rather due to the card being particularly effective when sent during the month of October.
Figure 3. Proportion of individuals who had vaccinations within 21 days of receiving the postcard reminder by treatment month.
Finally, we use regression to estimate the effect of the postcard 21 days after the postcard reminder was sent for the October, November and December groups. Table 6 reports these exploratory analyses, which suggest that the effect of the postcard is larger and statistically significant in October, smaller but still statistically significant in November and smallest in December. This result does not appear to be due to the length of time that the individual has had a postcard reminder in their possession. In the ‘Discussion’ section below, we return to this particular result and use it to suggest some hypotheses for future research.
Table 6. Effect of postcard reminder on vaccination rates within 21 days after release date.
*p < 0.1, **p < 0.05, *** p < 0.01.
October observations = 104,260.
November observations = 104,236.
December observations = 104,267.
Regressions included block indicators.
Discussion
Increasing vaccination rates among the elderly is an important goal for the Louisiana Department of Health and the CDC. This study shows that a simple and relatively low-costFootnote 14 intervention – sending reminder postcards to elderly individuals who are (over)due at least one vaccination – can modestly increase vaccination rates.
We sent behaviorally informed postcard reminders to 208,867 residents of Louisiana aged 65–70 who were listed as (over)due at least one of four vaccinations in the Louisiana IIS. The postcard had a small but detectable effect on rates of vaccination: at the end of the four-month trial period, the share of recommended vaccinations received was 0.27 percentage points higher in the treatment group compared to the control group; this translated to an additional 563 vaccinations. Further analysis of the data revealed that this treatment effect was primarily driven by additional flu and shingles vaccinations received by individuals who were sent the postcard in the month of October.
More broadly, our study serves as an example of applied research in settings where withholding a treatment from some study participants is not feasible. A stepped wedge design allowed us to rigorously evaluate the intervention while also allowing all participants to receive the treatment by the end of the study. Specifically, by staggering the postcard reminders such that they were sent out in different months and by randomly assigning individuals to receive the card in different months, we were able to estimate the effect of sending out the postcard and to compare the effect of the postcard across different months.
These results have practical applications and also point toward new hypotheses. Practically, the study demonstrates that a reminder intervention sent by mail can be effective at changing vaccine uptake among an elderly population who are due one or more vaccinations. The success of the intervention suggests that at least some of the hypothesized barriers – awareness/complacency and strong recommendations from health care providers – were slightly alleviated by the postcard. However, the modest effect size underscores the importance of considering cost–benefit calculations before implementing a reminder program. In this study, we do not have the requisite data to carry out a cost–benefit analysis, but we would encourage such analyses to accompany reminder programs wherever feasible.
Additionally, the estimated effect in our study is lower than that reported in previous studies. Better understanding the conditions (demographics, season, delivery format and frequency of the reminder) that may affect the efficacy of reminders is an important avenue of research, potentially especially suited for future meta-analyses of existing trials across multiple settings. The modest effect size also points to the potential relevance of access to transportation, a barrier that was identified as important in this population, but not addressed by the intervention. It is plausible that the severity of this practical barrier reduced the efficacy of the behavioral intervention, and increasing awareness may not matter if no transportation solutions are available. The modest success of the postcard thus simultaneously illustrates the promise and the limitations of behavioral interventions: they may be most useful in settings where practitioners have accurately identified behavioral barriers as a primary barrier. We encourage future researchers to carefully reflect on the context in which a behavioral intervention is considered; interaction effects between different barriers are likely to be important in determining the success of any given intervention.
This study also points to new hypotheses. We find that the postcard is effective for flu and shingles vaccinations (but not pneumonia or tetanus vaccinations) and only during the month of October. One reason for this finding could be a heightened awareness of vaccines in October. October, of course, is a particularly active month for campaigns that encourage people, especially the elderly, to get the seasonal flu vaccine. The Louisiana Department of Health ran a roster of outreach programs, which included free flu vaccines as well as an infant immunization campaign, during this period. Our data support this interpretation by showing that there are more vaccinations in October than in the other months.
This interpretation raises the possibility that reminders may be particularly effective if they are sent in the context of other related campaign activity or during times of otherwise heightened public awareness of vaccination requirements. In other words, vaccination campaigns may have a ‘the sum is greater than the whole of its parts’ quality, with different pieces of a campaign reinforcing each other to create and maintain the salience of vaccination among the public. The fact that the reminder is most effective for the flu vaccination in particular further reinforces the interpretation that external cues about the importance of flu vaccines may have interacted with our reminder.
While we are not aware of any other outreach campaigns for the shingles vaccine in Louisiana during this time period, we cannot rule out the notion that the ‘spillover’ effect that we observe for the shingles vaccine in October may be related to such campaigns, if they happened at the same time as our intervention. We encourage future researchers to consider, even if informally, the broader persuasion/outreach environment that their interventions take place in. To study how a postcard intervention could be affected by the persuasion environment, study designs with multiple sites that have different persuasion environments but similar outreach programs may be particularly relevant, as these would allow for the identification of interaction effects between outreach initiatives.
The heightened awareness of the flu after November 2017 could be another reason why October was the most effective month. The 2017–2018 flu season was particularly deadly and contagious. The CDC reported that influenza-like illness activity began to increase in November 2017 (CDC, 2019). Baton Rouge General Medical Center reported a 450% increase in positive flu tests compared to the same time in the previous flu season. Addressing this development, Louisiana Department of Health offered free rounds of flu vaccines due to the severity of the flu season (Lussier, Reference Lussier2018). This postcard intervention may have had the biggest effect in October 2017 because the flu had not started to spread yet. Once the flu season was underway and more people learned about how deadly that year's strain was, it is possible people began to vaccinate regardless of receiving a postcard reminder. Therefore, the postcard could have had no effect after October because, by November, everyone in Louisiana was encouraged to vaccinate. While this expectation would be difficult to study in a one-off study, meta-analyses of year-on-year outreach effects, in conjunction with data on the severity of the flu seasons, may be able to speak to it. More modestly, information about the severity of the flu season could be added as a randomized component to interventions when appropriate, measuring whether such information ‘cancels out’ the efficacy of other intervention language.
It is unclear why the spillover effect did not extend to pneumonia and tetanus vaccines, but one potential explanation is that the latter two vaccines have more complicated recommended uptake schedules compared to the seasonal flu vaccine and the one-off shingles vaccine. Future studies could explore both the seasonal and potentially cumulative effects of reminders and potential spillovers between different vaccinations. As we suggested for influenza outreach above, this type of research would likely require collaborations between researchers and institutions that run multiple interventions at once. Such collaborations would allow designs where several interventions are randomized at once, developing a systematic sense of the recipients’ information environment. Alternatively, to study spillover effects between different vaccinations within one site, a study could randomly assign a reminder regarding multiple vaccines or a reminder regarding one vaccine. The between-conditions difference in vaccines received would measure spillover between different vaccinations. To study the cumulative effect of reminders, a study could compare first-time recipients of postcard reminders with returning recipients. Strategies for identifying first-time recipients could leverage age thresholds, changes in eligibility requirements and moving from one region to another.
Finally, our study also has important limitations. The participants in this study were a particular subset of individuals: residents of Louisiana, aged 65–70, who may have been (over)due one of the four target vaccinations. Whether the results of this study are more broadly applicable – to other demographic groups, other forms of reminders or other vaccinations – is a question for future research. We also recognize that some subset of our participant pool likely received the postcard for the second year in a row; due to data limitations, we are unable to explore whether this affected the efficacy of the intervention, but considerations of repeated exposure will be important in future research as feasible.
Another important limitation arises from the difficulties of using a state IIS as a source of data. The information system in Louisiana holds a comprehensive list of the target population, but because reporting on adult vaccinations is not mandatory, vaccine records almost certainly represent an undercount of real vaccination rates. For our study, this means that our estimated effect sizes are likely biased downward, and that real effects may be larger. Future research with alternative data sources could address this shortcoming. Finally, the modest effect sizes in this study also point to the importance of considering behavioral interventions in their broader context: as discussed in the ‘Introduction’ section, behavioral interventions are most effective for addressing psychological barriers, but are less obviously applicable to practical barriers such as cost and accessibility. We encourage future researchers to pay attention to the environments in which behavioral interventions may be more/less effective in causing behavior change.
In sum, in accordance with the predictions of behavioral science, we found that a postcard reminder can modestly boost vaccination uptake among recipients. By including a stepped wedge design and block randomization in our evaluation of the study, we were able to compare the effect of the intervention in different months and for specific vaccines. We found that this intervention appeared most effective early in the flu season (in October), and that it boosted uptake of the flu and shingles (but not tetanus or pneumonia) vaccines.
Acknowledgments
The authors would like to thank Caglayan Baser, Ekrem Baser, Kelly Bidwell, Amira Boland, Jake Bowers, Pompa Debroy, Tony Fox, Christopher Grady, Michael Gusmano, Alice Iannantuoni, Jessica Leight, Elana Safran, conference discussants, anonymous reviewers and team members at the Office of Evaluation Sciences in the General Services Administration, the Immunization Program Office in the Louisiana Office of Public Health and the National Vaccine Program Office in the Department of Health and Human Services. The findings were previously presented at the annual conventions of the Midwest Political Science Association (2018), the American Psychological Association (2018), the Association for Psychological Science (2019) and the American Society of Health Economists (2019).
