Rapid assessment of influenza vaccine effectiveness: analysis of an internet-based cohort

Summary

The effectiveness of influenza vaccination programmes is seldom known during an epidemic. We developed an internet-based system to record influenza-like symptoms and response to infection in a participating cohort. Using self-reports of influenza-like symptoms and of influenza vaccine history and uptake, we estimated vaccine effectiveness (VE) without the need for individuals to seek healthcare. We found that vaccination with the 2010 seasonal influenza vaccine was significantly protective against influenza-like illness (ILI) during the 2010–2011 influenza season (VE 52%, 95% CI 27–68). VE for individuals who received both the 2010 seasonal and 2009 pandemic influenza vaccines was 59% (95% CI 27–77), slightly higher than VE for those vaccinated in 2010 alone (VE 46%, 95% CI 9–68). Vaccinated individuals with ILI reported taking less time off work than unvaccinated individuals with ILI (3·4 days vs. 5·3 days, P<0·001).

INTRODUCTION

Many countries routinely vaccinate against influenza. However, as vaccines must be developed before the influenza season, mismatches between the circulating and vaccine strains can occur. It is therefore seldom clear during the season how effective the vaccine is at reducing the risk of death, serious illness, influenza-like illness (ILI), laboratory-confirmed disease, absenteeism, or other adverse events.

Measuring vaccine effectiveness (VE) can be challenging because of the many potential confounders and biases. The ‘gold-standard’ approach involves following large matched cohorts of vaccinated and unvaccinated individuals over the course of a season and directly measuring the protective effect of the vaccine. In practice, such cohort studies are rarely used as they are both time-consuming and expensive. Instead, a range of alternative techniques has been developed to estimate VE more rapidly and with fewer resources [1–4].

Commonly reported techniques include the screening method and the case-control method [1, 2]. The screening method compares vaccination levels in influenza cases that are seen by the health services with estimated vaccination levels in the population as a whole, whereas the case-control method compares vaccination levels in ILI cases that are laboratory confirmed as influenza with ILI cases that test negative for influenza. These methods were used to estimate the efficacy of the H1N1v (2009) vaccine during the 2009–2010 season to be 70% in England and Scotland [5], and around 90% in Germany [6] and Canada [7], and have been applied elsewhere in previous influenza seasons [3, 4, 8]. The first estimates of the efficacy of the 2010 seasonal influenza vaccine, obtained using the case-control method, were recently published [9], suggesting an efficacy of around 63% in people vaccinated in both the 2009–2010 and 2010–2011 seasons.

Although widely used, these methods suffer from the fact that they require cases to present themselves for detection, generally to general practitioners (GPs), and to be identified as ILI cases and tested for influenza. It is highly unlikely that individuals with ILI who are tested for influenza by their GP are a random sample of individuals with ILI [10]. In particular, such efficacy estimates depend on assumptions about the relative rates of treatment-seeking and case-detection in vaccinated and unvaccinated individuals. There are a number of possible ways in which these assumptions may not be met: incidence of influenza-negative ILI may differ between vaccinated and unvaccinated groups; there may be differences in the presentation of symptoms between vaccinated and unvaccinated individuals, leading to differences in the probability of cases being identified for testing by GPs; there may be differences in viral shedding levels or duration of infectiousness leading to differences in test sensitivity. These reasons provide motivation for seeking additional methods of estimating VE that do not rely on individuals presenting themselves for treatment.

Here, we present a novel method for monitoring VE in real-time using an internet-based cohort [11, 12]. In contrast to other methods, internet-based surveillance does not rely on individuals seeking medical attention [10]. During a normal influenza season, internet surveillance can complement traditional methods by providing rapid preliminary estimates of VE and may prove vital during a pandemic when traditional surveillance methods are not practical [11, 12].

METHODS

The internet-based UK flusurvey (www.flusurvey.org.uk) [11] was launched for the 2010–2011 season on 1 December 2010. Participation was voluntary, and members of the public were encouraged to participate through outreach activities and the media. Participants completed an intake questionnaire on registration, recording their age, gender, risk group status (taking regular medication for diabetes, asthma, other lung, heart or kidney disorders, being immunocompromised or pregnant), whether and when they were vaccinated, public transport use, and contact with groups of patients, the elderly, and children. Each week, participants were prompted by email to complete a survey documenting any symptoms experienced, onset date, use of health services, and whether the episode resulted in absenteeism. Further details can be found in Tilston et al. [11]. Participants were able to update their reported vaccination status throughout the season, and were periodically reminded of this in the weekly email.

We analysed all reports collected up to 31 January 2011 from all participants who had completed the symptom questionnaire at least once. The study period, 1 December 2010 to 31 January 2011, includes the time during which GP-reported levels of ILI peaked and were above baseline levels (Fig. 1).

Fig. 1. Vaccine uptake in flusurvey participants. Cumulative 2010 seasonal influenza vaccine uptake among flusurvey participants across three key groups: participants aged ⩾65 years (red line, circles); participants aged <65 years in risk groups (blue line, squares) and healthcare workers (purple line, diamonds). Bars represent 95% confidence intervals. Also shown (grey line) is ILI incidence (weekly cases per 100 000), as reported by GP-based surveillance [18]. Date is shown on the horizontal axis. The launch date of flusurvey for the 2010–2011 season is indicated with a cross (×).

Episodes of illness reported through flusurvey were classified as ILI if the recorded symptoms matched the Health Protection Agency (HPA) ILI case definition of a fever plus two or more of: cough, sore throat, runny nose, joint pain, headache, vomiting or diarrhoea. Episodes of acute respiratory illness that did not match the ILI case definition but included symptoms of sneezing, cough, runny/blocked nose or sore throat were diagnosed as milder respiratory illness (MRI).

An unadjusted estimate of VE was estimated from the ratio of attack rates in vaccinated and unvaccinated participants as 1 – RR, where RR is the relative risk of reporting illness in the vaccinated and unvaccinated groups. As this crude estimate of VE is not adjusted for any covariates, such as age or belonging to a risk group, multivariate regression models were used to model the probability of ever reporting ILI and, for those individuals who did not report ILI, ever reporting MRI during the study period. Age group, membership of a risk group, contact patterns and vaccine status were used as predictor variables. For the purposes of this study, participants were considered as vaccinated during the 2010–2011 season if they reported having received a vaccine at any stage during the season. Because of the timing of the study, the majority of those vaccinated had received their vaccine before the study began (Fig. 1).

VE is defined as the relative reduction in risk between the vaccinated and unvaccinated groups. Because ILI is relatively common, Poisson regression with robust standard errors was used to estimate directly the RR of reporting ILI given each of the predictor variables [13] (the common approach of using the odds ratio (OR) to approximate RR is only appropriate when measuring rare events [13]). Estimates generated using log-binomial regression [13, 14] and adjusting ORs using the method of Zhang & Yu [15] gave very similar results (not shown). Analyses were conducted using R version 2.10 [16] and Stata v. 11 [17].

RESULTS

In all, 653 individuals registered for the flusurvey, of whom six were excluded from the analysis because of unknown vaccine status. The symptom questionnaire was completed a total of 3102 times, with a median of five reports per participant.

Table 1 shows the characteristics of the 647 individuals included in the analysis. Sixty-two percent of participants were female and, as might be expected from an internet-based survey, children and people aged ⩾65 years were underrepresented, with 78% of participants aged between 25 and 64 years. Influenza-like-illness (ILI) was reported at least once by 185 (27%) of the sample; MRI was reported on at least one occasion by 390 (60%) individuals, and by 309 (67%) of those who did not report ILI. Thirty-four percent (220) reported receiving an H1N1v-containing vaccine either in 2009–2010 (as the ‘swine flu’ vaccine) or in 2010–2011 (as the seasonal flu vaccine). Vaccine uptake during the 2010–2011 influenza season was 70% (95% CI 57–82) in participants aged ⩾65 years, with the majority vaccinated before November 2010 (Fig. 1). Participants aged <65 years with an underlying health condition reported a slower and slightly lower vaccine uptake of 60% (95% CI 49–70). Thirty percent (95% CI 18–47) of healthcare workers (defined as those who reported having regular contact with patients) reported receiving the 2010–2011 vaccine, with an increase in uptake coinciding with the rise in cases in December 2010 [18].

Table 1. Multivariate regression analyses of factors related to risk of acquiring ILI or MRI

ILI, Influenza-like-illness; MRI, mild respiratory illness; RR, relative risk; CI, confidence interval.

† ILI defined as fever plus two or more of runny or blocked nose, sore throat, cough, headache, muscle pain, vomiting, diarrhoea.

‡ MRI defined as ‘not ILI’ plus at least one of runny or blocked nose, sore throat, cough, sneezing. Participants reporting ILI were excluded from this analysis, so the relevant denominator is given by (N-N_ILI).

§ Defined as taking regular medication for diabetes, asthma, other lung, heart or kidney disorders, being immunocompromised or pregnant.

¶ Contact with more than 10 during the course of a typical day.

* P<0·05, ** P<0·01, ⁺P<0·1.

Participants aged >25 years were significantly less likely (P<0·05) to report ILI than younger participants (Table 1). Participants receiving the H1N1v vaccine either in 2010–2011 alone or both in 2010–2011 and 2009–2010 were significantly less likely (P<0·01) to report ILI than unvaccinated participants. Other factors were not statistically significant, although there was a trend for an increased risk of ILI in individuals who were in a risk group, or had regular contact with groups of children. In those individuals who did not report ILI, the risk of MRI was not associated with any factor except vaccine status: those reporting H1N1v vaccination in 2010–2011 and 2009–2010 were significantly less likely (P=0·029) to report MRI.

ILI was reported by 14% (26/181) of individuals vaccinated against H1N1v influenza in the 2010–2011 season and 34% (159/466) of unvaccinated individuals, giving a crude estimate of VE of 58% (95% CI 39–71). Adjusting for other factors, and considering vaccination in 2010–2011 only, VE against ILI was estimated to be 52% (95% CI 27–68). Additionally considering vaccination in 2009–2010, VE against ILI was estimated to be 46% (95% CI 9–68) for individuals who received the 2010 seasonal vaccine only and 59% (95% CI 27–77) for individuals who received both the 2010 seasonal vaccine and the 2009 pandemic vaccine. In those individuals who did not report ILI, VE against MRI was estimated to be 25% (95% CI 3–42) for those receiving H1N1v-containing vaccines in both the 2009–2010 and 2010–2011 seasons. There was no significant protection against illness during the 2010–2011 season afforded to individuals who just received the 2009 pandemic vaccine, although the number of such participants was small.

Vaccination was associated with reduced absenteeism among participants. In participants aged between 25 and 64 years, 4·1% of vaccinated participants and 11·6% of unvaccinated participants reported taking time off work due to symptoms (P<0·001). Of the participants who took time off work, vaccinated participants took an average of 3·4 days off and unvaccinated participants took 5·3 days off (P<0·001).

The healthcare-seeking behaviour of participants with symptoms did not vary by vaccine status after accounting for age and risk groups (RR 0·99, P>0·5).

DISCUSSION

The analysis presented here demonstrates the effectiveness of H1N1v vaccination at reducing reported ILI and at reducing absenteeism. We found VE at reducing ILI to be 59% in those receiving H1N1v vaccination in both 2009–2010 and 2010–2011. The results are broadly consistent with a randomized control trial that found that vaccination protected against ILI, physician visits, and laboratory-confirmed influenza in healthy adults when the vaccine and circulating viruses were well matched [19] and with estimates of influenza vaccine efficacy made using data from previous seasons [4–8]. Our results also agree with the first published study looking at the effectiveness of the 2010 seasonal vaccine in the UK [9], which reports an effectiveness of 46% for individuals vaccinated in 2010–2011 only and 63% in individuals vaccinated in both the 2009–2010 and 2010–2011 seasons, and with studies conducted elsewhere in Europe [20, 21] (we note that both our estimates and those in other studies [9, 20, 21] have fairly wide confidence intervals).

The finding that individuals vaccinated with both the 2009 pandemic influenza vaccine and the 2010 seasonal influenza vaccine may have had slightly elevated effectiveness against ILI and were protected against milder illness is consistent with results recently published elsewhere [9], and suggests that receipt of the pandemic vaccine in 2009–2010 may have primed individuals leading to a better response to the seasonal vaccine in that year. Further studies are needed to confirm or refute this.

There are several limitations to the approach taken here, many of which are discussed elsewhere [11]. The participants were a self-selecting sample with potential biases, and in particular the youngest and oldest age groups were underrepresented. ILI was determined using a symptom-based case definition, and therefore influenza was not laboratory confirmed; however, ILI as determined here has been shown to provide a good match to patterns of influenza incidence [10–12] and the difference in estimated effectiveness of vaccination against ILI compared to effectiveness against MRI suggests that ILI is capturing true influenza infections. In those individuals who did not report ILI, H1N1v vaccination offered some protection against MRI, suggesting that some MRI cases were actually mild doses of influenza. The approach here relies on self-reported symptoms and vaccination status, which, owing to the anonymous nature of the survey, cannot be verified. Furthermore, although participants were reminded to keep their vaccination status updated, we may not have recorded all vaccinations that took place after the study began; however, because the study started before the majority of ILI cases but after the majority of vaccinations had taken place in the UK (Fig. 1) [18], it is unlikely that many vaccinations went unrecorded. In the analysis presented here we have treated individuals as vaccinated if they received an H1N1v influenza vaccine at any point during the 2010–2011 season, hence ‘non-effectiveness’ of vaccination could be the result of infection taking place before vaccination; however, because of the timing of the 2010–2011 UK influenza season (incidence peaking in late December/early January) [18] almost all vaccinated individuals had received their vaccination before being exposed to influenza, and in our study there were only two individuals who reported ILI prior to their date of vaccination.

There are advantages to our approach. First, the internet-based system collects ILI incidence data almost in real time, and thus allows us to make extremely rapid estimates of VE; second, the approach follows vaccinated and unvaccinated individuals through the influenza season rather than relying on people with ILI accessing healthcare services, and is therefore free from potential biases introduced by GP-based estimates of vaccine efficacy; third, our community-based approach is inexpensive and can be scaled up to include more participants at no extra cost. Thus, while our study is relatively small, involving only about 650 participants, it is hoped that substantially larger sample sizes will be obtained in future years as the UK flusurvey becomes more established.

In common with methods that rely on people visiting their GP, there is no guarantee that our sample is representative of the population. However, among the participants in this study the level of vaccine uptake across various groups closely matches provisional data from the HPA (according to provisional data contained in the HPA weekly national influenza report [18], 72·8% of people in England aged ⩾65 years had received the 2010 seasonal influenza vaccine vs. 70% in our sample; in people aged <65 years in a risk group the comparable figures are 50·3% and 60%, respectively; and in healthcare workers provisional data show that 34·2% were vaccinated by the end of January 2011 vs. 30% in our sample). Thus, although there may well be other biases in our cohort, it resembles the UK population in terms of vaccination behaviour. Furthermore, there is no reason to expect that vaccinated and unvaccinated participants who exhibit flu-like symptoms would complete the survey in different ways. That traditional methods of estimating vaccine efficacy and the method presented here are obliged to make use of potentially biased samples is an unfortunate reality; we believe that the method described here, although in need of further testing over future seasons, is a valuable addition to existing techniques.

Internet-based monitoring of influenza in the community allows real-time surveillance of ILI, rapid assessment of VE, and an understanding of the wider impact of illness, which can be used to inform healthcare providers and the public of the utility of influenza vaccination in both resource-rich and resource-poor settings.