For and Against Brexit: A Survey Experiment of the Impact of Campaign Effects on Public Attitudes toward EU Membership

Abstract

What are the lessons of the 2016 referendum on UK membership of the European Union (EU) regarding the effects of message framing? This article reports findings from an innovative online survey experiment based on a two-wave panel design. The findings show that, despite the expectation that campaign effects are generally small for high-salience issues – such as Brexit – the potential for campaign effects was high for the pro-EU frames. This suggests that within an asymmetrical information environment – in which the arguments for one side of an issue (anti-EU) are ‘priced in’, while arguments for the other side (pro-EU) have been understated – the potential for campaign effects in a single direction are substantial. To the extent that this environment is reflected in other referendum campaigns, the potential effect of pro-EU frames may be substantial.

Referendums on a country’s relations with the European Union (EU) are relatively common. The 23 June 2016 referendum on whether the United Kingdom should remain in or leave the EU was the 55th such referendum in Europe since 1972, and the 30th since 2000. In some respects, the UK’s ‘remain-or-leave’ referendum was similar to other recent EU referendums.¹ The vote was triggered by a party in government that was divided on Europe. Whereas most centrist voters generally supported the pro-EU side, most voters on the radical right and a significant portion on the radical left supported the anti-EU side. The referendum was also preceded by growing public support for a radical right and ‘hard’ Eurosceptic party, the UK Independence Party (UKIP).²

In other respects, though, the 2016 UK referendum was different. Whereas most other EU referendums have been on reforms of the EU treaties, or on whether a country should join the EU or adopt the euro single currency, the 2016 referendum was only the third on whether a country should remain in or leave the EU. The only other such ‘high-stakes’ referendums were an earlier referendum in the UK, in 1975, when 67 per cent voted to stay in the then European Communities, and the Greenland referendum in 1982, when 54 per cent opted to leave the European Communities. Following the UK 2016 referendum, though, we may see similar in/out referendums in other member states. Several radical right parties in other EU states are campaigning for a ‘Cameron-style’ referendum, for example.

We use the 2016 referendum to achieve two broad goals. First, we test theories of campaign effects in a context that is extraordinary from the perspective of most persuasion and framing studies, which typically focus on low-salience issues.³ This may be explained by the fact that persuasion and framing affects are expected to be weak for long-standing issues on which individuals have been given the time and information necessary to form and solidify their opinions.⁴ Little opinion change is expected for crystallized attitudes.⁵ Leaving or remaining in the EU is the type of high-salience issue for which we would expect crystallized attitudes and weak campaign effects. Even before the referendum, voters considered the issue of Europe and the UK’s EU membership to be the leading issue facing the country.⁶ However, the UK referendum also represented a situation where substantial campaign effects were possible, because of the information asymmetry that existed prior to the campaign. On the one hand, before the referendum campaign, consumers of UK media had been exposed to at least forty years of Eurosceptic coverage.⁷ Public concern over immigration and dissatisfaction with established politics had been targeted by UKIP, which won the 2014 European Parliament elections and secured nearly 13 per cent of the vote in the 2015 general election.⁸ On the other hand, arguments regarding the positive effects of EU membership were far less prominent as they received a fraction of the media attention. This created a situation in which argument frames that emphasized previously under-stressed considerations might have had an effect despite the high salience of the issue. It is particularly unusual that positive – but not negative – messages had the potential to influence opinions. Negativity bias usually leads to greater mental attention to negative arguments relative to positive ones, leading to larger persuasion and framing effects.⁹

Persuasion and framing effects in the context of the 2016 referendum may also have been larger than expected due to weak partisan cuing. When an issue is highly salient, each political party typically takes a clear position. When this happens, it is difficult to move partisans away from their partisan-consistent position. Hence, both persuasion and framing effects are expected to be small or non-existent. Yet the 2016 Brexit referendum represented an unusual situation in which cues from the two major parties were unclear. Longstanding divisions within the incumbent Conservative Party were reflected in the fact that while 185 Conservative MPs campaigned for ‘Remain’, 138 declared support for ‘Leave’. This divide extended to the very top of the party, as at least six Cabinet ministers campaigned for Leave, including the Mayor of London and Conservative Member of Parliament (MP) Boris Johnson and Justice Secretary Michael Gove, alongside the UKIP leader, Nigel Farage, and ten Labour MPs.¹⁰ While an overwhelming majority of Labour MPs were remain supporters, the party leader, Jeremy Corbyn, was lukewarm in his support, both because he was known to be critical of the EU and because he did not want to be seen to support the Cameron government. As a result, at the beginning of the campaign, which is when we undertook our study, the Labour leadership’s position was unclear.

In sum, the context of the 2016 EU referendum in the UK allows us to test persuasion and framing effects in a context in which the high salience of the issue would typically lead us to expect little to no effects, particularly for positive arguments, but the information asymmetry and the weak partisan cues of the environment lead us to expect strong effects. This is an atypical situation in which to test persuasion and framing effects, and leads to some unique theoretical expectations. Empirically testing these expectations provides a strong test of what we believe we know about campaign effects.

Even though the context of the Brexit referendum is extraordinary from the perspective of testing campaign effects, it does have some features in common with possible future in/out referendums. This allows us to achieve a second goal: to study the power of arguments used in the UK referendum to shed light not only on what happened in 2016 but also on what could happen in similar referendums in the future. In particular, our context allows us to examine whether relatively unfamiliar ‘pro-EU’ arguments that stress the benefits of EU membership can sway public attitudes in the face of well-established ‘anti-EU’ arguments that have been prominent in the UK and other EU nations.

We pursue these two goals via an online survey experiment, conducted between September and November 2015, just before the official referendum campaign began. The experiment included two innovative elements. First, it was designed to allow us to test the effects of different combinations of real-world arguments against each other during an actual campaign on public support for leaving/remaining in the EU. Secondly, the experiment was based on a ‘before and after’ design, using a two-wave panel, which allows us to examine how individuals changed their opinions in response to a set of arguments. In the next section we set out our theoretical expectations, after which we explain the design of our experiment and present the results. We conclude with a discussion of the implications of our results for the campaign effects literature and for future referendums.

Our main findings are that: (1) ‘pro-EU’ arguments on their own (could have) significantly increased support for remaining in the EU while ‘anti-EU’ arguments, which for decades have dominated the political and media debate about EU membership, and which we suggest were therefore ‘priced in’, had less potential to impact support for either side; (2) the effect of pro-EU arguments was mitigated when contrasted with anti-EU arguments, limiting the potential effectiveness of pro-EU arguments in an environment of competing pro-/anti-messages; (3) the effect of both pro- and anti-EU arguments was greatest among those whose prior beliefs were consistent with the argument and (4) the effect of pro-EU arguments was greatest among those with lower political knowledge (less education).

THEORY

Media, politicians and campaigns can portray information in different ways, stressing certain evaluations or parts of an argument.¹¹ Our research on the effect of different arguments on EU membership is rooted in existing research on persuasion, and how the ‘framing’ of issues can affect public opinion and vote choices generally, as well as research on how such framing can shape attitudes towards the EU specifically.

It is well established in political science that frames select and organize information on issues, give meaning, attribute positive or negative values, and can influence attitudes. Frames supply a readily comprehensible basis on which to think about the issue and how to justify what should be done about it.¹² A framing effect occurs when individuals who are exposed to different frames arrive at different positions on the issue, depending on the priority given to various considerations in the frame; effects can also be persistent.¹³

When we consider the effects of different arguments that are made for remaining in or leaving the EU, we view these arguments as frames and are interested in the effects of the different considerations raised in the different frames.¹⁴ There are a number of psychological processes that determine whether a consideration has an effect on opinion. In testing the effects of different arguments, we manipulate two of these: availability and accessibility.¹⁵ For a consideration to have an effect, it must be available in an individual’s memory and it must be accessible (front of mind) to be applied to the current situation.

Framing studies typically argue that they manipulate the accessibility of different considerations by presenting research participants with different arguments.¹⁶ However, there exists a debate about whether these framing studies manipulate the availability or accessibility of considerations. That is, whether exposing individuals to frames makes new information available or makes information that was already available to them accessible.¹⁷ The effect of the latter (accessibility) is typically considered a framing effect and that of the former (availability) a persuasion effect. We remain somewhat agnostic about this distinction, and argue that when we expose respondents to our frames, this has the potential to make both new information available and pre-existing information accessible. Given the salience of the issue of remaining in or leaving the EU leading up to the referendum, we expect that most arguments were already available and our arguments primarily manipulated accessibility. In other words, any detected effects are framing effects. However, given the pre-existing asymmetry in the volume of claims made regarding the negative and positive consequences of remaining in the EU, it is possible that the pro-EU arguments increased availability. In other words, pro-EU arguments may have also produced a persuasion effect.

Consistent with a campaign in which two camps advocate diametrically opposed positions (that is, whether to remain in, or leave, the EU), we view the relevant arguments as ‘valence frames’. As Lecheler and de Vreese observe, this valence alludes to one of the most fundamental aspects of political discourse, namely that political elites try to influence public reactions to an issue by emphasizing the positive or negative aspects of it, such as the positive or negative features of EU membership.¹⁸ As past studies observe, with regard to framing and the EU, ‘[t]he basic finding of two decades of research is that political communications that frame European integration in terms of its benefits increase public support for European integration, while messages that frame it in terms of costs, risks or cultural threats lower support for European integration’.¹⁹

Indeed, a growing number of experimental studies investigate frames as they relate to EU issues and how they affect public attitudes. Abbarno and Zapryanova randomly exposed respondents in the UK and Bulgaria to a ‘cultural threat’ or ‘democratic deficit’ frame, and found that both reduced support for EU integration.²⁰ Schuck and de Vreese demonstrate how news media frequently use ‘risk’ and ‘opportunity’ frames when discussing the issue of EU enlargement, frames that are shown to have significant effects on citizens that are, in turn, moderated by political knowledge.²¹ Studying attitudes toward Turkey’s possible EU membership, de Vreese et al. show how negative frames yielded stronger effects than positive frames, though more politically sophisticated citizens were more affected by positive frames, providing some evidence that attitudes can be shaped by the framing of the issue in interaction with individual characteristics.²² Overall, pro- and anti-EU economic, cultural and political frames have the potential to increase or decrease support for remaining in the EU.

The magnitude of a framing effect depends on the context. We argue that one way in which context matters is that if certain considerations are already highly available and accessible to the individual, exposure to frames that contain those considerations will have little effect. Under such circumstances, those considerations are already ‘priced in’ – already accessible and available. This is particularly relevant to a situation in which one side of an argument may have dominated the dialogue for an extended period of time. Given Britain’s long tradition of Euroscepticism, we expect that new pro-EU frames that are not ‘priced in’ to the debate will have a greater effect on vote choice than older and more familiar anti-EU frames. On that basis, we hypothesize that pro-EU frames will increase support for remaining in the EU while anti-EU frames will have little effect.

Another way in which context could matter is particularly relevant to a campaign in which two opposing sides make competing claims. The effect of a frame may differ depending upon whether the individual is simultaneously being exposed to competing frames. When individuals receive competing frames, they may choose the side that is consistent with their prior beliefs.²³ However, if an individual contrasts competing frames, this can cause him or her to take an intermediate position. The effect of one frame is mitigated by the opposing frame, resulting in weak effects.²⁴ We anticipate that both of these effects are relevant. We hypothesize that exposure to competing frames mitigates the effects of framing. At the same time, frames will have stronger effects if they are consistent with prior beliefs.²⁵ Therefore, we hypothesize that pro-EU frames will be more effective among people who previously expressed positive EU sentiment and/or those who identify as Labour supporters, while anti-EU frames will be more effective among people who previously expressed negative EU sentiment and/or who are Conservative identifiers.²⁶

The magnitude of a framing effect also depends on other characteristics of the recipient of the frame. Consistent with extant research, we expect that exposure to frames will have stronger effects on the less knowledgeable by making considerations available and accessible that were previously not accessible, something found in studies of framing and EU issues.²⁷ Among the more knowledgeable these considerations are likely to already be highly available and accessible, and so little effect is expected from exposure to frames that contain these considerations. Lecheler and de Vreese²⁸ find evidence of this type of moderating effect for knowledge when examining the effects of news framing on support for EU enlargement. Using education as a proxy for knowledge,²⁹ we hypothesize that framing effects will be stronger among the less educated.

This consequently leads us to the following hypotheses:

1. 1. Anti-EU frames, which are likely to already be highly available and accessible (priced in), will have less of an effect than pro-EU frames.

2. 2. In a context in which there are competing frames, the effect of each frame will be mitigated, resulting in weak effects.

3. 3. Frames consistent with prior beliefs will have stronger effects.

4. 4. The effect of frames will be greatest on those with less knowledge/education.

EXPERIMENTAL DESIGN

We test these hypotheses in an innovative, online population-based experiment that combines the strengths of laboratory experiments and national surveys.³⁰ By randomly assigning a sample of the adult population to different treatments we can identify the causal effects of being exposed to different types of arguments on support for EU membership. Conducting our experiment on a sample designed to reflect the national population, we can also estimate the magnitude of these effects on the population as a whole.³¹

The experiment was designed to simulate the ‘real-world’ campaign environment by exposing respondents to a combination of different frames, in the form of vignettes, about the UK’s EU membership, and which they might realistically receive in the referendum campaign, due to start shortly after our experiment. Our experiment reflects the real world in two ways. First, we use vignettes that include competing frames with different vignettes placing different weights on the two sides of the argument. Chong and Druckman³² note that there is little literature on the effect of competing frames. Sniderman and Theriault, and Brewer and Gross³³ examine competing frames but only with equal exposure to both. Of course, in reality arguments are rarely equal in exposure. Secondly, our experimental vignettes were inspired by real-world arguments used by the two sides of the debate only a few months before the referendum. Following past research on the effect of information on EU attitudes,³⁴ our vignettes include cultural, economic and political messages. Also following past research³⁵ these messages were based on leaflets, speeches or websites associated with the pro-EU/Remain or the anti-EU/Leave campaigns, thereby maximizing the generalizability of the results to the referendum. The arguments were tailored so that the length of each was approximately equal with roughly equivalent amounts of information. The primary variation across treatments was the direction of the argument (that is, pro-EU or anti-EU) rather than the amount of information. Our vignettes purposely did not include the source of the considerations presented. This allowed us to estimate the effects of the considerations uncontaminated by cueing from the source. The vignettes are shown in Appendix I.

Our experiment used a 2×2×2 full factorial design, in which each respondent was exposed to a vignette composed of: (1) a ‘pro’ or ‘anti’ EU cultural frame; plus (2) a ‘pro’ or ‘anti’ EU economic frame; plus (3) a ‘pro’ or ‘anti’ EU political frame. One-ninth of respondents were assigned to each of the eight possible treatment groups and one-ninth was assigned to a control group, which was not exposed to a vignette.³⁶ Because every treatment contains three frames, we are able to estimate the impact of different combinations of arguments holding the number of frames constant.

The full factorial design allows us to estimate the effect of: arguments containing frames all in the same direction (all pro- or all anti-EU); arguments with competing frames (some pro- and some anti-EU); and arguments with different weights placed on the two sides (a varying number of pro-EU and anti-EU arguments). This, in turn, allows us to test our hypotheses regarding the effects of different types and combinations of frames. The experiment was conducted before the date of the referendum was confirmed and the conclusion of EU reform negotiations, hence minimizing external contamination.

The population-based experiment was conducted as a two-wave panel study conducted by YouGov. The first wave was conducted between 24 September and 1 October 2015 and the second wave was conducted two weeks later between 16 October and 9 November 2015. The first wave included 6,074 completions. With a retention rate of 87.8 per cent, the second wave sample included 5,333 respondents, with approximately 590 respondents per group.³⁷ Further information about the sample, including balance tests, can be found in Appendix II. Once data were collected we proceeded with a difference-in-differences analysis. This allows us to estimate the average change in referendum vote intention before and after exposure to a vignette for individuals in the eight treatment groups (the eight argument combinations) compared to the change in referendum vote intention for individuals in the control group.

In the first wave, vote intention was measured using a standard referendum voting intention question: ‘How would you vote if the referendum on Britain’s membership of the EU were held tomorrow? (Remain in the EU/Leave the EU/Don’t Know)’. In the second wave, respondents were assigned to one of the vignettes (eight treatment groups) or to the control group. Respondents were then immediately asked the same vote intention question.

The panel structure allows us to measure referendum vote intention before and after exposure to the vignettes. This provides us with a within-subject design, allowing us to estimate the change in opinion, for each individual, due to exposure to a vignette relative to those who were not exposed to a vignette (the control group). For example, we estimate the average change in referendum vote intention for individuals exposed to a vignette with three pro-EU frames compared to the change in referendum vote intention for individuals in the control group. We also did this for individuals exposed to vignettes with the following combination of arguments: two pro/one anti, one pro/two anti and three anti. The within-subject design also provides greater power to determine the effects than a between-subjects design.³⁸

The two-week period between Waves 1 and 2 was designed to prevent respondents from thinking about their reported vote intention (in Wave 1) when reporting their vote intention (in Wave 2) after exposure to the vignette. This reduced the probability of suppressing the effect of a vignette as respondents attempt to appear consistent. Contrasting the change for each treatment group against a control allows us to control for any universal changes in vote intention in the intervening two weeks.

RESULTS

In the first wave, 38.2 per cent of respondents indicated they would vote to remain in the EU, 43.9 per cent indicated they would vote to leave, and 17.9 per cent indicated that they did not know or would not vote. In the second wave, among those in the control group (who were not exposed to any arguments), these results changed slightly to 37.1 per cent, 44.0 per cent and 18.8 per cent, respectively. To illustrate the magnitudes of the shifts across the two waves, Appendix Figure A1 shows the ‘flows’ from Remain/Leave/Don’t Know in Waves 1 and 2 for the control group and the two most extreme treatment groups: those exposed to three pro-EU arguments and those exposed to three anti-EU arguments.

To quantify and test the statistical significance of the average change in referendum vote intention before and after respondents were exposed to a particular combination of the arguments, we estimate the following:

$$\Delta vote_{i} \, {\equals}\, \alpha {\plus}\beta treatment_{i} {\plus}{\epsilon}_{i} $$

where $\Delta vote_{i} $ is the change in vote intention from Waves 1 to 2 for respondent i; and $treatment_{i} $ is a binary variable equalling 0 if the respondent was in the control group and 1 if the respondent was exposed to that particular combination of arguments. Respondents exposed to a different combination of arguments were not included in this particular estimate. Separate equations were estimated for changes in vote between Remain and some other vote (Leave or no vote), and for changes in vote between Leave and some other vote, and between no vote and some other vote. In each case, the coefficient β is the estimate of the average change in vote intention for those exposed to the combination of arguments, relative to the control group. This is our estimate of the effect of the vignette. Doing this for each of the eight possible combinations of arguments gives us an estimate of the effect of each of the eight vignettes.

Figure 1 plots the estimated effects of each vignette on Remain and Leave vote intention (the full model estimates are provided in Appendix V).³⁹ Exposure to three pro-EU arguments produced a statistically significant (p<0.05) increase in support for remaining in the EU and a decrease in support for leaving the EU. The increase in the Remain vote was 5.2 percentage points and the decrease in the Leave vote was 3.6 percentage points. There was no corresponding statistically significant effect for the three anti-EU arguments. None of the vignettes that contained any combination of anti-EU arguments produced a statistically significant effect on referendum voting intention. Meanwhile, in addition to a positive effect from the three pro-EU arguments, the ‘pro-cultural, pro-economic, anti-political’ vignette produced a statistically significant change in the vote. It increased public support for Remain and decreased support for Leave by 3.1 percentage points. Other vignettes that included pro- and anti-EU arguments did not produce a statistically significant effect.

Fig. 1. Difference-in-differences effects of the treatments Note: the figure shows the effect of a treatment on the change in support for Remain (Leave) relative to the change in support for Remain (Leave) in the control group. The full results are in Appendix Table A4.

Having determined that some vignettes that included pro-EU arguments produced statistically significant results, we then specifically estimated the effects of different vignettes depending on the number of pro- versus anti-EU arguments. We separately estimated the effects of all vignettes with three pro-EU arguments, vignettes with two pro- and one anti-EU argument, vignettes with one pro- and two anti-EU, and vignettes with three anti-EU arguments. Doing so allowed us to further examine the differential effect of pro- and anti-EU frames, and to compare the effect of exposing respondents to non-competing frames (all negative or all positive) to that of exposing respondents to competing frames with different weights placed on each side of the argument. Figure 2 reveals again how the vignette with three pro-EU arguments produced a statistically significant effect, as did vignettes with two pro- and one anti-EU arguments (our full results are reported in Appendix V). These latter vignettes increased public support for remaining in the EU by a statistically significant 2.3 percentage points. Their effect on support for Leave was not statistically significant. Overall, vignettes that were predominantly pro-EU in tone had a statistically significant effect on referendum vote choice, while vignettes that conveyed anti-EU messages did not.⁴⁰ This is consistent with our first hypothesis.

Fig. 2. Difference-in-differences effects of the relative volume of arguments Note: the figure shows the average effect of the relative volume of Pro/Anti EU arguments on the change in support for Remain (Leave) relative to the change in support for Remain (Leave) in the control group. The full results are in Appendix Table A6.

Does this mean that anti-EU arguments had no effect? We cannot reject the null hypothesis that anti-EU arguments had no effect, while we can reject the null hypothesis that pro-EU arguments had no effect. To test whether this is because anti-EU arguments have no effect, rather than because we do not have the power to detect their effect, we provide a test of the following null hypothesis: vignettes with two or more pro-EU arguments have no more of an effect (in the positive direction) than those with two or more anti-EU arguments (in the negative direction). This is a test of the difference in the magnitudes of anti-EU vignette effects against the pro-EU vignette effects. At the 0.05 level (one-sided), we can reject this null hypothesis for the proportion indicating that they would vote Remain (z score = 0.046; p = 0.04).⁴¹ For the Remain vote, we have evidence that the magnitude of the effect of pro-EU arguments was greater than the effect of anti-EU arguments and, in fact, no evidence that anti-EU arguments had any effect. Overall, this is strong evidence for our suggestion that anti-EU arguments (on average) had no effect while pro-EU arguments did.

Result 1: consistent with our first hypothesis that priced-in considerations will not have an effect, anti-EU frames have less of an effect than pro-EU frames.

Figure 2 also reveals that while a purely pro-EU vignette affected the proportion indicating they would vote Remain or Leave, a vignette containing two pro-EU frames and one anti-EU frame had less of an effect. Further, a vignette containing only one pro-EU frame and two anti-EU frames had no effect. The effect of the pro-EU frames was reduced when competing anti-EU frames were combined in the vignette.⁴²

Result 2: consistent with our second hypothesis, when there are competing frames, the effect of each frame is mitigated, resulting in weak effects.

Next, to examine the effect of prior attitudes on the effectiveness of pro- and anti-EU frames, we conducted two tests. First, we examined how effects differed among those who previously declared moderate feelings towards the EU, those who said they held extremely negative feelings and those with extremely positive feelings. The left panel of Figure 3 plots the distribution of EU attitudes by those who indicated that they intended to vote Remain, Leave or Don’t Know in Wave 1. Attitudes were measured on a 0 to 10 scale with 10 indicating the most positive and 0 indicating the most negative attitudes towards the EU. The negative end of the spectrum (0–3) was almost entirely populated by those who intended to vote Leave and the positive end of the spectrum (7–10) was almost entirely populated by those who intended to vote Remain. The moderate values (4–6) were a mixture of all three positions, with the centre dominated by those who did not know how they would vote.

Fig. 3. Extremists and moderates Note: the left-hand figure shows the distribution of respondents on a 0–10 Anti-/Pro-EU scale. The right-hand figure shows the average effect of each group of treatments on the change in support for Remain/Don’t Know/Leave relative to the change in support for Remain/Don’t Know/Leave in the control group. EU moderates were coded at points 3, 4, 5, 6, 7; Anti-EU extremists at points 0, 1, 2, 3; and Pro-EU extremists at points 7, 8, 9, 10. The full results are in Appendix Table A12.

On the basis of this distribution, we defined three groups of individuals based on their EU attitudes: anti-EU extremists, EU moderates and pro-EU extremists.⁴³Figure 3 presents the estimated effects of the vignettes on these three groups of individuals. We see that, as before, anti-EU arguments did not have any effect. However, either three or two pro-EU arguments increased the Remain vote among pro-EU extremists and EU moderates but not among the anti-EU extremists. This increase in the Remain vote came largely through a decrease in those indicating they did not know how they would vote. As expected, those with priors consistent (or at least not inconsistent) with the frames were most affected by them.

Somewhat counter to expectations, there was some effect of the pro-EU frames on anti-EU extremists. This effect was to decrease the Leave vote by increasing those who did not know how they would vote. What we are observing is a limitation on the effect of a frame depending on the strength of the individual’s priors. Those who had the most negative EU attitudes could be moved from a Leave vote to an undecided position, and those with the most positive EU attitudes could be moved from an undecided position to a Remain vote. However, individuals with extreme (pro- or anti-EU) attitudes were unlikely to be moved all the way from a Remain to a Leave vote or vice versa.

Our second test of how prior attitudes moderate the effectiveness of pro- and anti-EU frames examined how the effects of the vignettes differed by partisan identity. Figure 4 Panel A shows the effects of the vignettes on respondents who self-identified as Conservative or Labour Party identifiers. Conservative identifiers were not affected by pro-EU arguments; they were only moved away from Remain and towards Leave by the combination of three anti-EU messages. Labour identifiers, however, were moved by pro-EU arguments but not anti-EU arguments. The combination of either two or three pro-EU arguments moved Labour supporters away from saying they don’t know how they would vote and towards voting Remain. This is consistent with our expectations that those who identified as Labour would have prior attitudes that made them more susceptible to pro-EU frames, and that those who identified as Conservative would have prior attitudes that made them more susceptible to anti-EU frames.

Fig. 4. Difference-in-differences effects by subgroup (A). By party support (B). By age when finished education Note: each figure shows the average effect of each group of treatments on the change in support for Remain/Don’t Know/Leave relative to the change in support for Remain/Don’t Know/Leave in the control group. The full results are in Appendix Table A13.

Finding 3: in support of our third hypothesis, frames consistent with prior beliefs have stronger effects.

Finally, we examined education as a moderator of the effects of the vignettes. We made the assumption that those with higher levels of education would, on average, have high levels of political knowledge.⁴⁴ As shown in Figure 4, Panel B, we found that three or two pro-EU arguments increased the Remain vote only among those with the lowest levels of education (<17 years).⁴⁵ There were no other effects for the higher levels of education, except for a decrease in Remain and an increase in Leave due to three anti-EU arguments among the middle education category (17–19 years). To the extent that education correlates with level of political knowledge, it appears as though those with lower levels of political knowledge were more likely to be affected by frames that made information available and accessible.

Finding 4: consistent with Hypothesis 4, the effect of both pro- and anti-EU frames is strongest for those with the least knowledge/education.

CONCLUSIONS

In this article we have sought to build and expand upon existing research on persuasion and framing effects, on public attitudes toward the EU, voting behaviour and referendums. Our experimental design allowed us to test for the effect of ‘real-world’ arguments on both sides of the referendum campaign that attempted to influence the vote through an array of pro-EU or anti-EU messages. Our main finding is that, in one of the most Eurosceptic states in the EU, pro-EU arguments had the potential to significantly increase support for remaining in the EU, while anti-EU arguments had less potential to impact support for either remaining or leaving. Our results suggest that in more recent years the well-rehearsed arguments about the perceived costs, risks and threats from the EU became ‘priced in’ to the national debate about continued EU membership. Having experienced several decades of a Eurosceptic tabloid press, an established current of Conservative Euroscepticism and the rise of the Eurosceptic radical right UKIP, it is perhaps not surprising that participants in our survey were less strongly influenced by prominent Eurosceptic arguments than hitherto less well-known messages about the benefits of EU membership. In contrast, our results suggest that communicating the claimed benefits and advantages of EU membership to an electorate that had long been noted for its instinctive Euroscepticism might have had a significant impact on the overall vote.

In terms of testing theories of campaign effects, our results suggest that even though campaign effects are generally expected to be small when the issue(s) at stake is(are) highly salient, under conditions of information asymmetry coupled with low partisan cuing, the potential for a campaign effect in a single direction is substantial. The one-sidedness of the potential effect is important because it is under these circumstances that a campaign can have a decisive effect on the outcome. Our results are also important because they demonstrate that the high salience of the issues at stake in a campaign does not necessarily mean that all information is already available and accessible and that the electorate is immune to the way in which an argument is framed. Important campaign effects are still possible in referendums on highly salient issues.

In terms of shedding light on the power of arguments used in the UK referendum and similar referendums in the future, our results suggest that certain groups might have been especially receptive to pro-EU frames, notably Labour Party supporters, people under 26 (see Appendix VI), people with lower levels of education, and those undecided about their EU referendum vote. Our results also suggest that pro-EU arguments weakened opposition to the EU among the most fervently committed Eurosceptics and increased support for Remain among more moderate voters. While we find little evidence that pro-EU arguments resulted in ‘direct switching’ between the two camps, our results suggest that the campaign to keep the UK in the EU held important advantages, if it had focused on a clear and consistent pro-EU message.

Given that public attitudes toward EU membership were highly polarized and finely balanced before the referendum, such that the way the respective cases were framed could have made all the difference, our results suggest that it might have been a mistake for the Remain campaign to focus primarily on the potential economic costs of leaving the EU – which the Leave campaign dismissed as ‘Project Fear’ – rather than making a positive case for remaining in the EU. That said, it is important to note that the effectiveness of pro-EU arguments is mitigated when they compete with anti-EU considerations. Therefore, a pro-EU campaign would have had to dominate the anti-EU message in order to be effective. Further, it must be acknowledged that a Conservative Party leadership that had been critical of the EU for three decades could not have credibly led such a strong pro-EU Remain campaign. This credibility trap might be absent in other EU member states facing possible in/out referendums, where elites could perhaps make pro-EU economic, political or cultural arguments. However, anti-EU arguments may still be effective in countries that have not experienced the UK’s long tradition of Euroscepticism.

Overall, an important implication of our work is that in the context of information asymmetry and weak partisan cues, it is valuable to identify which arguments have become commonplace and which have not. Commonplace arguments may be effective at holding public opinion in place, but will be ineffective at shifting opinion during a referendum campaign. Arguments that have been typically understated have the potential to deliver a substantial impact.