Large-Scale Evidence for the Effectiveness of Partisan GOTV Robo Calls

Abstract

We document the effectiveness of automated (robo) calls for increasing voter participation in contrast to most published research which finds little or no effect from automated calls. We establish this finding in a large field experiment which mimics campaign behavior with a targeted, partisan get-out-the-vote campaign. Our findings show that across all treatments, automated calls led to three additional votes for every thousand subjects called during the 2014 midterm general election. Additionally, our experimental design allows for testing how the number of calls in a treatment, that is dosage, affects voter turnout. Here, results show that three extra calls increase the treatment effect to seven additional votes per thousand subjects called, but that too many additional calls decrease that effect to statistical insignificance in a six-call treatment.

Introduction

We conducted a large-scale experiment to assess the effectiveness of partisan robo calls. ¹ We find a positive, significant treatment effect of increasing voter turnout. Our experiment has several key features that stand out in the get-out-the-vote (GOTV) literature. First, our experiment varies the intensity of the treatment dosage by using a different number of calls (one, three, or six) for different treatment groups. We found the largest treatment effects for subjects scheduled to receive three calls. Second, a Republican-leaning media firm administered the experiment and targeted likely Republican voters. In contrast, many similar published experiments are nonpartisan, Democratic, or otherwise left-leaning. Third, the subject target of our experiment was potentially marginal voters. Specifically, people who had voted in two of the previous four elections. These three key features may explain why this experiment shows that robo calls have a positive, significant treatment effect on voter participation, while many other robo call experiments do not. In this section, we will discuss important results from the robo call GOTV literature and highlight several important features of the experiment and how they relate to previous GOTV studies.

Previous research shows that the impact of robo calls on voter turnout is small and not statistically significant. This is summarized in a meta-study by Green, McGrath, and Aronow (2013). They report a statistically insignificant increase in voter turnout of 0.156 percentage points, attributable to prerecorded phone messages. Another meta-analysis reports an “average intent-to-treat effect of 0.113 percentage point, with a 95% confidence interval ranging from −0.336 to 0.563” (Green and Gerber 2015, p. 196), also implying a statistically insignificant effect of robo calls on voter participation. ²

One study that did find a significant positive treatment effect of robo calls is Gerber et al. (2010), which used a placebo design to increase the precision of the estimate of its treatment effect. ³ Studying the 2008 primary election, Gerber et al. (2010) find an approximately two percentage point increase in voter participation for subjects who were successfully contacted, relative to subjects in a placebo group who were successfully contacted. The authors attribute the large size of their reported effect to call content, or “social pressure” messaging, which reminded subjects that they had voted in the past two general elections but not in the past primary, and informed subjects that their voting records are publicly available. However, when attempting to replicate their results for the November 2008 general election in a follow-up study, the authors find “a weakly positive effect in the November 2008 general election, in keeping with the usual pattern of weaker turnout effects in high-salience elections” (Green and Gerber 2019, 89). Ali and Lin (2013) develop a model of voter turnout based on similar social dynamics. DellaVigna et al. (2014) present evidence from a field experiment that “individuals vote because they expect to be asked” about it and find lying costly. That research suggests that understanding the dynamics of social pressure messaging amidst an evolving campaign landscape is a fruitful research agenda.

Our experiment does not use social pressure messaging but does find a statistically significant positive effect for robo calls placed before the November 2014 general election. Subjects in the combined treatment group were 0.318 percentage points more likely to vote than subjects in the control group – or a rate of 1 mobilized voter for every 314 subjects called. The treatment with three calls was the most effective. Subjects in the three-call treatment were 0.695 percentage points more likely to vote than subjects in the control group – or a rate of 1 mobilized voter for every 144 subjects called.

We posit that robo calls are most effective when they are received by voters who are on the margin as to whether to participate or not to participate in voting. That is, a GOTV treatment will be less effective if it is primarily targeted to individuals who are habitual voters or individuals who never vote. Most of the individuals with a high propensity to vote will vote regardless of the treatment. Similarly, individuals with a very low propensity to vote are unlikely to be mobilized by a robo call. However, an individual’s propensity to vote is context-dependent and can change based on the salience of a given election to that potential voter (Arceneaux and Nickerson 2009). Moreover, that an individual’s propensity to vote is context dependent is consistent with an array of rational choice models of voting (Ali and Lin 2013; Coate and Conlin 2004; Degan and Merlo 2011; Dhillon and Peralta 2002; Feddersen and Sandroni 2006). Thus, along several dimensions, such as the choice of states without multiple high-profile statewide races and the subject selection criteria, the experiment was designed to include a higher proportion of subjects who were likely to be marginal voters.

We built a sample of potentially marginal voters by focusing on registered voters who had voted exactly twice, during the 2010 and 2012 primary and general elections. Thus, these voters missed voting in two of the four elections. Other GOTV studies use alternative criteria for subject selection. For example, Ramirez (2005, p. 70) selects subjects from a list of “registered Latino voters in low-propensity precincts” Other studies select subjects that are expected to be receptive to the treatment message, which is in a similar spirit to our experimental design. For example, Shaw et al. (2012, p. 236), whose calls include an endorsement by Texas Governor Rick Perry, direct their calls to subjects that are “both likely primary voters and strong Perry supporters.” Gerber et al. (2010) select only subjects who voted in the past two general elections but did not vote in the most recent primary.

As noted above, our experiment is targeted at Republican voters, while many other robo call and other GOTV experiments target Democratic or nonpartisan voters. Examples of such experiments include, e.g., Nickerson and Rogers (2010), Barton, Castillo, and Petrie (2016), and Rogers, Green, Ternovski, and Young (2017) which are all targeted to Democrats (with messages of varying degrees of partisanship) while Ramirez (2005), Nickerson (2008), Gerber, Green, Kaplan, and Kern (2010), and Panagopoulos (2011) had nonpartisan targeting. Among studies of robo calls, Shaw, Green, Gimpel, and Gerber (2012) is an exception; it targeted voters in a Republican primary.

It is possible that differences between partisan groups or differences in messages could cause particular GOTV methods, such as robo calls, to be more effective for one voter group than another. This effect could possibly be mediated by voters’ age or other voter characteristics. We do not have prior expectations that robo calls are more (or less) effective when targeting Republican voters as opposed to Democratic or nonpartisan voters, especially after controlling for individual characteristics. However, we believe that it is important to close the empirical gap in studies of the effectiveness of robo calls with experiments that have Republicans in general elections as their main subject pool. This approach will give insights into whether findings from experiments targeting Democratic and nonpartisan voters generalize to Republican voters.

One of the important features of this experiment is the use of treatment groups that vary in the number of GOTV robo calls received. With a few exceptions, published GOTV experiments use a single robo call, while it is common for political candidates and advocacy groups to deploy multiple iterations of multiple types of GOTV communication. Ramirez (2005) included a treatment with two robo calls among a wide assortment of GOTV methods. Recently, Zelizer (2020) studied the effect of repeating robo calls, dividing approximately 40,000 registered voters into eight treatments that received zero to seven calls in the week leading up to a primary election. Zelizer found that a treatment with five calls was the most effective, but the differences between treatments were not statistically significant. We are not aware of any other studies that directly measure the effect of robo call dosage.

Similarly, Green and Zelizer (2017) conducted an experiment using multiple treatments with GOTV mail and found that efficacy declined after five mailers. Other experimental studies have raised the issue of GOTV over-saturation (Gerber and Green 2000) or analyzed GOTV synergies via multiple treatments (Green and Gerber 2019, p. 184), but did not find an effect of such multiple treatments. In a manual for political operatives, Lofy (2005, p. 153) recommends two to nine contacts before an election to increase the likelihood that habitual voters cast a ballot on the day of the election.

Procedures

The field experiment occurred in the days leading up to the November 2014 general election. The experiment was funded and conducted by a political consulting firm seeking to evaluate the effectiveness of their GOTV services. ⁴ The firm identified registered voters in six states as subjects. The potential subject pool consisted of likely Republican voters who had registered to vote in Georgia, Nebraska, New Mexico, Ohio, Pennsylvania, or Virginia prior to January 1, 2010. ⁵ The pool of potential subjects was narrowed to people who had voted in exactly two of the 2010 and 2012 primary and general elections. People who voted early or who cast an absentee ballot in the 2010 or 2012 general election were also excluded.

From this group, 42,000 subjects with unique landline phone numbers were randomly selected from each state. Many of these landline numbers are associated with households that have more than one voter. Because we cannot identify who answers a robo call, we included every registered voter in each selected household in the analysis even though these household members are not expected to all match the original selection criteria in terms of voting history or even party affiliation. The 42,000 households from each state yielded subject pools ranging from 86,714 to 95,557 registered voters. In total, the study includes 539,567 subjects.

Within each state, households were randomly assigned to one of four groups: a control group and three treatment groups. Subjects in the three treatment groups, T1, T3, and T6 received one, three, or six treatment calls over 1, 3, or 6 days, respectively. The robo calls to subjects in T6 started 6 days before the election, the calls to subjects in T3 started 3 days before the election, and all subjects in the three treatment groups received a call on the morning of Election Day. On average, each household has 2.16 registered voters. Detailed information, including summary statistics of subject characteristics in each treatment group, is in Appendix A. The table of summary statistics shows that there are some imbalances across treatments on some control variables, particularly 2010 and 2012 voting rates. To address this issue, we include these covariates in our regression models. Further, we cluster the standard errors by household in all household-level specifications.

Treatment call messages were slightly different on each call date. Each message included a reminder about the date of the election and a short partisan message encouraging the subject to vote. Most of the messages invoke negative partisanship rather than making specific promises or describing specific plans. ⁶ The duration of the messages was between 35 and 45 seconds. Appendix B lists the script for each message. Table 1 shows the schedule of calls and schedule of scripts, which were identical for every state.

Table 1. Treatment Call Schedule

Note: Households in the treatment groups were called in 2014, on the date listed in the first row of Table 3. The general election was on November 4, 2014. Appendix B includes the text for each of the six scripts.

We define “live answerers” as subjects residing in households where at least one call resulted in a live answer, meaning someone in that household answered the phone for any length of time. We classify all other subjects as “treatment non-answerers,” regardless of whether the call went to an answering machine, operator, no answer, busy, fax machine, or was otherwise uncompleted. ⁷ However, calls that went to answering machines or had a live answer were considered successful treatment for the purposes of the local average treatment effect (LATE) analysis described in Appendix E, because subjects may have listened to and been influenced by the message left on their answering machine. Our outcome variable – whether a subject voted in the election – is based on verified voting records for the November 2014 election.

Results

Table 2 shows that the percent of subjects with at least one live answer for T1, T3, and T6 were 39, 60, and 67%, respectively. ⁸ The remaining subjects in each group did not answer any of the treatment calls. Table 3 shows the mean number of calls with live answers for subjects in each treatment group. ⁹ This pattern of more live answers in treatments with more calls is consistent across states.

Table 2. Treatment Call Outcomes by Treatment Group

Note: The table shows the percent of subjects in each treatment group that either answered at least one call live, had no live answer, had at least one answering machine answer, etc. We refer to these “responses” as “call outcomes.” Treatment call outcome categories, which are not all mutually exclusive, are listed in the rows of this table. Percents are relative to the total number of subjects in each treatment group. “AM” denotes answering machine. The raw number of subjects in each cell is given in parentheses.

Table 3. Mean Number of Live Answers by Treatment Group

Note: The top panel of this table shows the mean number of live answers for all subjects in each treatment group. The bottom panel shows the mean number of live answers only among subjects with at least one live answer. Standard errors for each mean are listed in parentheses.

Table 4 reports the voting rates for each of the treatment groups. These results show that treatment was associated with modestly higher voting rates, particularly the three call treatments, with voting rates among all treated subjects and subjects in T3 approximately 0.3 and 0.6 percentage points higher, respectively.

Table 4. Voting Rates by Treatment Group

Table 5 uses multivariate regressions to report the intent-to-treat estimates from comparing voting rates of the three treatment groups to those of the control group while controlling for subject-specific voting history, demographics, and socioeconomic characteristics. ¹⁰ Specifically, our control variables include whether or not a subject voted in the 2010 and/or 2012 general elections and subject age, gender, educational attainment, estimated income, state of residence, and the number of registered voters in their household. ¹¹ The coefficients on each of the treatment variables show the difference in voting rates, measured in percentage points, between the corresponding treatment group and the control group. ¹² Standard errors are clustered by household and listed below each regression coefficient.

Table 5. Intent-to-Treat Effects for All Subjects and Single-Voter Households (SVH)

Note: The table shows OLS estimates for the difference in voting rates between subjects in the treatment group and subjects in the control group. The subset of subjects used to estimate each specification is shown with the “Population” label. Subject controls include whether or not a subject voted in the 2010 or 2012 general elections or both and subject age, gender, education, income, number of subjects in the household, and state of residence. Standard errors are clustered by household and displayed in parentheses.

*** p < 0.01,

** p < 0.05,

* p < 0.1.

In Table 5, Column 1, the coefficient on the All Treatments variable is 0.00318, which indicates that voter participation for subjects in the pooled three treatment group is 0.318 percentage points higher than for subjects in the control group. This point estimate is statistically significant at the 5% level. The magnitude of this estimate indicates a yield of three additional voters for every 1,000 subjects who were called, or 314 subjects called per additional voter mobilized. ¹³ This effect is larger than the average robo call effect reported in Green and Gerber (2015, p. 196). This result implies a cost of under $9 to induce a subject from not participating in the election to casting a ballot. ¹⁴

Table 5, Column 2 reports estimation results by each treatment group. The findings indicate that the most effective treatment is T3, the treatment in which subjects receive an automated call on each of 3 days. T3’s treatment effect is 0.695 percentage points which is statistically significant at the one percent level. ¹⁵ This estimate implies the turnout of one additional voter for every 144 subjects assigned to T3. ¹⁶ The treatment effect for groups T1 and T6 are positive, but smaller and statistically insignificant. The implied cost to induce an additional vote for only subjects in T3 is under $4.

Our findings show that calling on 3 days was more effective than calling on 1 day or over 6 days. ¹⁷ The Wald statistic for testing for the equality of the coefficients for T1 and T3 is 9.04 and is significant at the one percent level. The Wald statistic testing the equality of the coefficients for T3 and T6 is 8.21 and is statistically significant at the one percent level. Thus, for both tests, we reject the hypothesis of equal dosage effects. There is no statistically significant difference between the coefficients for T1 and T6.

In households with multiple registered voters, we do not know which subject or subjects answered the treatment call or calls. Because the treatment effect may be diluted in multi-voter household, we also analyze subjects in single-voter households, which accounted for approximately one tenth of all subjects. ¹⁸ In Table 5, Columns 3 and 4 provide those estimation results. Our data show a larger treatment effect for subjects in single-voter households. In Column 3, the average treatment effect for single-voter households is 0.388 percentage points, but it is not statistically significant at the ten percent level.

Table 5, Column 4 separates the results by treatment group and shows the estimated effect among single-voter households. The treatment effects for T1 and T3 are both positive, although only T3 is statistically significant at the ten percent level. The estimate for T6 is negative, but not statistically significant.

As noted in the Procedures section above, the treatment messages were different from day to day. Within T3 and T6, subjects differed in terms of which days had live answers, and thus which messages were heard. However, we were not able to identify any meaningful differences in turnout that can be attributed to differences between hearing specific message scripts. ¹⁹ Similarly, although previous researchers find that GOTV calls are less effective the earlier these calls are placed prior to an election (Green and Gerber 2019, p. 186; Murray and Matland 2014; Nickerson 2007; Panagopoulos 2011), we did not observe a timing effect within the groups of voters who received three or six calls. This suggests that within the time frame of our study, the effectiveness of the treatment did not measurably decay.

We found that the pattern of treatment effectiveness varied across the states in our sample. We have included a chart in Figure 1 to show each group’s treatment effect for the six states in the experiment. While there are differences in the balance of subject characteristics across states, we do not observe a correlation between differences in some of the balance tests and the estimated treatment effects across the states. The state-by-state subject characteristic summary statistics are included in Appendix A as are the regression tables for the results graphed in Figure 1.

Figure 1. Intent-to-treat effects by state and treatment group.

The chart shows the state-by-state OLS estimates for the difference in voting rates between subjects in each treatment group and subjects in the control group in that state. Each circle represents the estimated treatment effect, and the vertical lines indicate the 95% confidence interval for each estimate. The regression used to calculate these treatment effects used subject controls, including whether or not a subject voted in the 2010 or 2012 general elections or both and subject age, gender, education, income, and number of subjects in the household. Standard errors are clustered by household.

In Appendix D, we discuss an alternative method of estimating the causal treatment effect using data from a call placed after the election. In Appendix E, we report the conventional method of estimating the treatment on treated effect by calculating local average treatment effects. In Appendix F, we test whether our results are significant after correcting for multiple hypothesis testing.

Conclusion

This experiment employed multiple-call dosage and mimicked the behavior of real political GOTV campaigns. We show that targeted automated calls have a positive effect on voter turnout even without using a social pressure message. We find that, across all treatments, the intent-to-treat effect is larger than most previous measures of robo call effectiveness. Our results suggest that it is not irrational for campaigns to deploy robo calls as a cheap additional tool to increase voter turnout. We find that the treatment increases voter participation by between three to six additional voters for every 1,000 subjects. This corresponds to a cost of $4 to $9 to induce a subject to vote in elections.

We also find that dosage matters. The intent-to-treat effect for dosage T3 is two to three times larger than for dosage T1, while dosage T6 typically had the smallest effect. These results suggest that additional calls can increase effectiveness, but that too many calls may be counterproductive. Our data are not rich enough to determine why the T6 treatment had the smallest treatment effect. One explanation for this finding might be that receiving so many calls annoyed the subjects and that they started ignoring the calls that were closer to the election, even if they were picking up the phone. While T6 reached a higher percentage of subjects than T3 – 67 and 60% – respectively, the increase from three calls to six calls nearly doubled the total number of live answers for each live answerer – from 2.1 in T3 to 3.8 in T6. Annoyance with this stark increase in live answers received may explain the decreased effectiveness of T6. This topic may warrant further experiments, using placebos, to determine how receiving marginal additional calls affects voting behavior.

Overall, the pattern of dosage results observed in our experiment is consistent with the broader GOTV literature. The 0.12 percentage point effect of T1 is consistent with meta-analyses (Green and Gerber 2015, p. 196; Green, McGrath, and Aronow 2013), ²⁰ while the larger 0.65 percentage point treatment effect of T3 suggests, along with more recent work (such as Zelizer 2020), provides evidence that robo call treatments are more effective with some, but not too much, repetition.