Ethics in Elite Experiments: A Perspective of Officials and Voters

Experimental research using politicians as subjects has become increasingly important to the field of political science (for overviews see Costa 2017; Desposato 2016; Grose 2014). During recent years, elite experiments have allowed scholars to gain insight into notoriously sensitive areas, such as politicians’ prejudices related to gender and race (Broockman 2013; Butler and Broockman 2011; McClendon 2016); the effects of campaign contributions on policy makers’ behavior (Kalla and Broockman 2015); and politicians’ personal responsiveness to public opinion (Butler, Karpowitz, and Pope 2012; Butler and Nickerson 2011; Öhberg and Naurin 2016). The rise of experimental studies using elite subjects has instigated within-the-field thoughts on research practices and ethics (see Loewen, Rubenson, and Wantchekon 2010; McClendon 2012). The pool of politicians is small and recruiting elite subjects to experiments is one of the foremost challenges for the field to solve. In a note on political science experiments in the journal Science, James N. Druckman and Arthur Lupia specifically raise the importance of elites’ participation in political science experiments:

Yet, important challenges persist in expanding the domain of experimental political science. One such challenge is that typical experimental subjects often lack the experience needed to act ‘as if’ they were professional legislators; yet, legislators themselves are often reluctant to participate in experiments as subjects. (2012, 1178)

This research note takes on the challenge to clarify politicians’ own perceptions of elite experiments. We make use of a research infrastructure in Sweden which gives regular access to large samples of politicians via online panel surveys. Recently, these surveys have introduced experimental designs and at the same time triggered curiosity about how the politicians actually regard the idea of being the subject of experiments. We believe that some of the lessons learned are of interest also to others in the field, hence this note.

We here use a research design which compares politicians’ and citizens’ perceptions of research ethics in general, and of elite political science experiments in particular. More specifically, we try to estimate whether politicians differ systematically from citizens in their views on research ethics surrounding elite experiments and we identify variation in perceptions of elite experiments within the group of politicians.

Politicians’ and Citizens’ Perceptions of Ethical Problems in Elite Experiments

To give a background to the study: It is a commonly held belief in studies using elite experiments that politicians are less sensitive to experimental designs than ordinary citizens because they are used to being scrutinized by the media, voters, opponents and others. The perspective that politicians can probably ‘take more’ is borne out in that IRB processes often are interpreted to put fewer ethical constraints on elite experiments than on other types of experiments (see Grose 2016; Malesky 2016; Schrag 2009).¹ One reason for this is that politicians willingly put themselves in situations where they are ‘tested’ and ‘mislead’. Their job is to be held accountable, so the harm that is done to them through manipulations in political science experiments can be regarded as small in comparison to the scrutiny they are accustomed to in the ordinary game of politics. In this sense, politicians are expected to be better prepared to deal with the discomfort and deception which come from experimental manipulation than others.

However, some scholars have called for a more thorough discussion on what it means to ‘do no harm’ to public officials (see Druckman, Leeper, and Mullnix, 2014, 206; Malesky 2016, 218). At least in theory, it should also be possible to make the other argument – that politicians are more sensitive to experimental designs than citizens. One could argue, for example, that politicians have more at stake than ordinary citizens do when they participate in experiments. Politicians might be more vulnerable than citizens are since their job is dependent on them being credible. If politicians can be manipulated in experimental research, then scholars can cause them more harm than would be the case with ordinary citizens. Furthermore, since the pool of elite subjects is much smaller than the pool of citizen subjects, such harm might be considerably more damaging to future research if it is done to politicians rather than to citizens (cf. Desposato 2016; Druckman, Leeper, and Mullnix 2014). Moreover, the stakes are higher when politicians are manipulated or deceived. Democratically elected representatives (or journalists, to use another example) have special tasks to perform in democratic societies. Politicians represent not only themselves, but also their constituents.

There are also reasons for expecting differences within the group of politicians, which is an important insight for a field that often suffers from skewed samples. For example, previous work has at times assumed that local and part-time politicians are more sensitive to experimental studies than other politicians. Local politics is often less ideologically polarized and less affected by the medialization and professionalization of politics. Local politicians can therefore be assumed to be less well accustomed to being ‘scrutinized’ and ‘manipulated’; they are more like ‘normal people’ (see Grose 2016; Malesky 2016). Work on citizens’ views of research ethics, furthermore, has found that women are generally more ethically concerned than men (Ameen, Guffey, and McMillan, 1996), indicating that similar differences could be found also among politicians. In a similar vein; having a higher education and being older may indicate a greater understanding of how to play the political game (Jerit, Barabas, and Bolsen 2006), thereby resulting in these politicians being more accustomed to, and possibly also less sensitive to, the consequences of a life in politics. There is also reason to believe that ideological differences can play a role in how useful political science research is perceived to be in general (see Lupia and Aldrich 2014; Rothstein 2016), which could affect the perceptions of research ethics.

Our note focuses on perceptions of survey experiments. A recent overview of experiments in political science research showed that survey experiments are the most often used type of experiments and that they constitute a swiftly increasing part of published political science research (Desposato 2016). Furthermore, survey experiments are useful in research on political and administrative elites as they demand fairly little effort from the subjects. What is more, they have proven useful to scholars when attracting generous public funding from national and European research councils. Taken together, there is a need to understand how elite participants regard their participation in these increasingly used survey experiments. More specifically, our aim in this note is to gain an understanding of (1) whether Swedish politicians view survey experiments as problematic, (2) whether they are more or less concerned compared to non-politicians and (3) whether there are differences within the group of politicians, when it comes to how survey experiments are perceived.

Study Design

Our data come from the Laboratory of Opinion Research (LORE), at the University of Gothenburg. LORE conducts opt-in university-run panel surveys of Swedish citizens and politicians. We received a randomly selected sample of n=5,960 out of a total of 64,656 respondents to the Citizen Panel. Our response rate was 63 per cent, providing us with 3,755 respondents. The survey took place between November 2015 and January 2016. Descriptions of the sample and how it compares to the entire Swedish population can be seen in Table A1 in Appendix 1. It shows that our sample of citizens is more educated than Swedes in general. There is also an over-representation of males. The Panel of Politicians consists of Swedish politicians from local, regional and national levels. Politicians are recruited to the panel through email addresses found on the websites of local, regional and national assemblies. Recruitment is also done by asking respondents of population-based surveys whether they would agree to take part in online political science surveys on a regular basis. More specifically, such requests were sent out to all respondents of the Comparative Candidate Study (CCS), which was distributed to all candidates running for the Swedish Parliament in 2010 and 2014. The same was done for respondents to a survey that went out to all politicians in local and regional assemblies in Sweden (Kommun- och landstingsfullmäktigeundersökningen, KOLFU 2012). The response rate of our survey (which was performed between November 2015 and January 2016) was 63 per cent, meaning 1,864 politicians with various backgrounds participated in our survey. Descriptions and comparisons with the entire population of politicians in Sweden are provided in Table A2 in Appendix 1. In short, those analyses show that our study provides results from a large and varied sample of actual politicians functioning in the studied context.

It is worth noting that we do not suspect that Swedish politicians delegate the answering of this survey to staff, not even on the national level. The personal administrative backup of Swedish politicians is in general small (and for local politicians often non-existent) (see Gilljam, Karlsson, and Sundell 2010). And the willingness of Swedish politicians to answer surveys is generally large (Karlsson and Nordin 2015). The opt-in format probably limits delegation of the answers even more. We also note, therefore, that this means that we study a case where politicians have comparably high trust in scholars to begin with, and that they are relatively willing to participate in studies. Hence, this is probably a case of trusting politicians who are used to functioning in an open and research-friendly system. Concerns over research ethics from this group should therefore be taken particularly seriously, as they likely begin with a positive approach to participating. We will come back to the generalizability of our results in our concluding section.

Survey

We approach our research questions from three angles. First, we ask politicians and citizens to tell us how they would feel if they were subjects in a survey experiment. (Since ‘survey experiment’ is not an often-used notion outside of the research community, we elaborate a bit on it to be as clear as possible, see below.) Second, to be able to compare how problematic survey experiments are found to be, we ask politicians and citizens about other, more obviously problematic, survey practices. Third, we perform a survey experiment ourselves where we ask politicians and citizens to evaluate a study where an experiment is performed on fictive respondents, and we vary whether these respondents are politicians or citizens.

Below is the survey battery that helped us achieve the first two angles.²

To what extent would you find the following things to be ethically problematic if you were asked to participate in a survey addressed to you/to you in your capacity as politician? ³

1. 1. The researcher allows an assistant to manage the communication with you as a respondent instead of handling it herself/himself.

2. 2. Your participation in the survey takes longer than what the researcher originally indicated.

3. 3. The questionnaire contains questions that are of a more private nature than what seemed to be the case when you agreed to participate.

4. 4. The survey includes experiments where randomization is used to give certain respondents one type of information and other respondents another type of information.

5. 5. The researcher does not have time to analyze the survey you participate in.

6. 6. The researcher does not provide a report of the results which you can access.

Respondents answered using a seven-point scale: 1 – Yes, very problematic, 4 – Neither problematic nor unproblematic 7 – No, completely unproblematic.

To achieve the third angle, we asked respondents to evaluate the ethics of an experiment which was described either as a ‘study’ or an ‘experiment’, and which had either ‘politicians’ or ‘citizens’ as subjects.⁴ More specifically, we used the following vignette:

Imagine the following situation. A researcher wants to investigate whether Swedish politicians/citizens are particularly friendly towards people with typically Swedish names. The researcher conducts an experiment/a study where a selection of politicians/citizens assesses a job applicant’s qualifications. Everyone assesses the same CV, but some of the politicians/citizens get to read a typical foreign name, while others read a typical Swedish name. According to you, is such an experiment/a study with politicians/citizens ethically problematic? (Response options: 1 – Yes, very problematic, 4 – Neither problematic nor unproblematic 7 – No, completely unproblematic.)

Results

Figure 1 gives an overview of the six different survey practices that respondents were asked about.

Figure 1 Citizens and politicians assess survey experiments and other ethically problematic practices of political science research Note: Item formulations are found in the text. Confidence levels at 95 per cent.

Figure 1 shows that citizens and politicians rank the ethical problems in the same way (the mean values are under control for age, education, gender and party). The Total bars illustrate that politicians are somewhat more troubled about ethical problems (mean 3.49) than citizens (mean 3.70), p<0.01. Out of the six practices, politicians find five of the items to be more ethically problematic than citizens do, in particular ‘contact with assistants’, ‘survey experiments’, ‘no analyses’ and ‘no reports’. The respondents do not differ in their views on the potential problem of a survey being ‘too private’. Looking at the individual items and the extent to which they are deemed to be problematic, we can see that the least problematic practice is allowing an assistant to take care of communication with a respondent. Thereafter comes manipulating respondents via experiments in the survey. Hence, according to our respondents, it is of greater importance to make sure that the data are analyzed, and that respondents receive something in return, such as a report of the survey results, than it is to abstain from survey experiments. However, the politicians are more troubled by survey experiments than are the citizens.

Moving on to the experiment on discrimination, we can see the same pattern emerge: respondents do not perceive it to be particularly ethically problematic, which we illustrate in Figures 2 and 3 (where the groups that saw the word ‘experiment’ and the groups that saw the word ‘study’ are collapsed⁵). The means for all groups are around 5 on the seven-point scale. However, we find no support for the idea that politicians are less sensitive to being subjects in experiments than are citizens. Politicians’ mean approval of the experiment with politicians (5.03) is not significantly different from their mean approval of experiments with citizens (4.96). Instead, as Figure 2 demonstrates, citizens are less approving of the experiment on politicians than they are of the experiment with citizens. The mean for citizens when the subject of the experiment is politicians is 5.08, and the mean for the same experiment with citizens as subjects is 5.30, p<0.05. Put differently: The politicians do not show more concern for citizens than for politicians, but citizens show more concern for politicians than for citizens.

Figure 2 Citizens on whether the fictional experiment is ethically problematic

Figure 3 Politicians on whether the fictional experiment is ethically problematic Note: See Figure 1. Confidence intervals 95 per cent.

Analyses so far indicate answers to our two first questions: (1) Politicians in the case of Sweden do not seem to find survey experiments particularly problematic, but (2) they are somewhat more concerned than our sample of non-politicians. To answer our third question – whether there are differences within the group of politicians when it comes to approval of survey experiments – Table 1 presents multivariate OLS regressions using the independent variables mentioned above and the item on survey experiments as dependent variable.⁶ We note that the independent variables come from our exploratory ideas, rather than from strict testing of hypotheses. This merits caution when interpreting levels of significance.

Table 1 Politicians’ approval of survey experiments (OLS)

Standard errors in parentheses ***p<0.01, **p<0.05, *p<0.1. The question formulation of the dependent variable is described in the text. The equivalent analysis for non-politicians is included in Table A6 in Appendix 1.

Table 1 indicates that politicians who have a university education are more likely to approve of survey experiments. Younger politicians are slightly more likely to approve than are older politicians. However, we detect no differences among men and women in terms of how survey experiments are viewed, nor do we find any differences between national MPs and local politicians. Variation is instead found among representatives of different political parties. The Christian Democrats, the Conservatives, the Social Democrats, the Center Party and the representatives of the nationalist party, the Sweden Democrats, are less likely to approve of survey experiments than are representatives from the Left Party, the Feminist Initiative, the Green Party and the Liberal Party.

Conclusions

The recent increase in elite survey experiments has not been backed up by empirical evaluations of how elite subjects perceive of their role in experiments. This research note gives some such back-up. Our conclusion is that, as political scientists, we probably risk alienating elite respondents more with lengthy surveys and the tendency to gather data that does not result in research reports, than by performing survey experiments on them. This is an important result and should help strengthen the field.

However, and this is important as Sweden is a case where politicians generally trust scholars, our results do not support the idea that ‘politicians can take more’ than citizens in social science studies. Rather, our data suggest that politicians are more concerned than citizens about research ethics. Our results therefore call for continued awareness. As elite survey experiments most likely will continue to be used, perceptions of research ethics among elite subjects will be dependent on future scholars’ continued respect and caution.

How do our results travel to other contexts? The case used in this note is fairly narrow and results cannot be assumed to travel easily. One reasonable guess is that Swedish politicians are less concerned compared to politicians in contexts where trust in social science research is comparatively lower. Such lower (and more polarized) trust in social science scholars is found in the US from where much of the current elite experimental research comes. This indicates to us that the concerns raised here (that politicians show more concern about research ethics than non-politicians) deserves caution also in other contexts. The same goes for politicians in countries with conservative or right-wing political traditions. There is an indication in our data of a liberal/conservative dimension in the perceptions of elite survey experiments as representatives of different parties express varying levels of approval. We note that the parties that distinguish themselves as comparatively less concerned are parties that score relatively high on the ‘Green, Alternative, and Libertarian’ values in the so-called GAL-TAN-dimension, and low on the ‘Traditional, Authoritarian and Nationalist’ values (Hooghe, Marks and Wilson 2002, 966; Lindvall et al. 2017).

However, the ranking of survey experiments vis-à-vis other survey practices might travel more easily, that is, the result that survey experiments are not perceived as more problematic than other practices. Furthermore, we find evidence of diverging (and non-diverging) views on research ethics in survey experiments among politicians which might travel to other contexts, where well-educated and younger politicians are less concerned about survey experiments compared to less educated and older politicians. We also note that there are no large differences between local and national politicians or between woman and men. Moreover, while our results come from one particular context and one specific point in time, the idea that politicians in Sweden perceive research ethics differently depending on education, age and party affiliation should be of interest for any researchers in the field who want to maximize participation in survey experiments and minimize systematic dropouts. On this point, we would also like to note that we found evidence that politicians differ on the other aspects of research ethics that we ask about as well (full results are presented in Appendix 1). And since elite subjects are becoming both increasingly important and increasingly inaccessible for survey research, it seems worthwhile for scholars to consider the potential of diversifying their survey strategies in order to secure higher response rates.

It is worth noting that we have not studied the real sanctions politicians would meet if they were manipulated in experiments. Future studies should look into questions like: Would voters mind if politicians are able to be manipulated or would they appreciate the politicians’ effort to contribute to research? When are politicians punished or rewarded for participating in research? Furthermore, while we contribute a first empirical test of perceptions of elite experiments, far from all types of experiments have been tested in this note. Most notably, while we here sought to understand perceptions of survey experiments, it is often elite field experiments that raise most concern among scholars (see Desposato 2016).

Lastly, we return to the fact that the problem deemed most important by the respondents – and here they agree – is that scholars do not always analyze the data they collect, nor do they report the results back to those who have contributed as respondents. This is food for thought for anyone working in survey research, especially in the field of experimental research, where studies are sometimes left in the desk drawer due to null results.