Experiments, Journals, and Ethics

As political science experiments have increasingly moved out of the traditional western university laboratory setting, researchers have had to confront many new issues that had previously received less attention from experimentalists. Much ink and effort has been expended on the methodological issues involved due to the loss of experimental control and the difficulty in implementing and maintaining true random assignment. Moreover, as experimentation has become increasingly popular and more attention has been paid to the way in which we collect and analyze data generally, many are concerned with disconnects between design proposals and actual implementations – especially with the possibility of burying null findings or reporting findings that just barely meet standardized levels of significance – and with the ability of researchers to replicate results. Furthermore, questions of harm to human subjects – and notions of informed consent – have had to be revisited in situations far from the conditions in which these concepts were originally developed.

Since becoming co-editors of JEPS we have realized that a number of activists on these fronts think one possible solution to many of these concerns is to have journal editors take on a larger monitoring role: by facilitating pre-registration of designs, participating in transparency movements, enforcing standards in reporting, checking for IRB reviews, and requiring authors to submit data sets for replication. In many cases we agree with the advocates. For example, in our first issue we published the report of the Experimental Research Section Committee on Reporting Standards and we have asked all of our authors to address these standards in their research. ¹ We also require authors to either provide an IRB approval for their research or else a written statement as to why such approval was not necessary, as well as to sign a statement documenting – and address in the footnotes of the article – any possible conflicts of interests in their research. However, in other cases, we do not agree. For example, at JEPS we have resisted the movement to review designs of experiments before they have been conducted with a promise of future publication because we have not been convinced (yet?) that doing so is advantageous, although we look forward to seeing the first application of this in practice in a political science journal (of which we are aware) in a forthcoming issue of Comparative Political Studies. ² We have also resisted suggestions to “rate” articles in terms of adherence to pre-registration designs for reasons that one of us has elaborated on elsewhere. ³

Nevertheless, up until recently less attention has been focused on the ethical concerns involved with moving out of the standard laboratory. That is, in the standard laboratory experimental setting subjects typically provide informed consent to participate in the study and are aware that they are involved in an experiment. The manipulations in the laboratory generally have minimal long term effects or effects on naturally occurring events or cross-effects on non-subjects. But experiments that occur outside the laboratory have the potential to affect larger numbers of individuals, most – and often all – of whom have no idea that they are participating in experimental research. Furthermore, these experiments may have the potential to have effects on people who are not even part of the study. Recently, however, concerns about the ethical implications of moving outside the laboratory have grown. And as these concerns have become more prominent, increasingly the journal review process has been seen as one way of enforcing ethical standards on social science researchers.

It is our view that such a demand on journals is appropriate. Why? It would be easy to pass on such an obligation. We could take the view that by the time the research reaches us it has already been conducted and thus our ethical concerns as editors are no longer relevant. Moreover, determining what is ethical is difficult, as there are many different opinions and standards across the discipline and countries. Yet, if we ignore ethical issues, then we implicitly encourage unethical conduct in the discipline.

It is our view that researchers do not conduct unethical experiments because they are unethical per se: that is, we believe that most researchers desire to conduct ethical experiments. But ethical experiments are, in many cases, more time consuming, and may lead to research that is harder to publish. They require researchers to think more carefully about how aspects of the design might lead to risks and harms that are beyond what they and their subjects experience in their everyday lives. ⁴ The design stage of experimental work is the most important stage in that the cleaner and more straightforward the design, the easier we can use the data to make subsequent causal inferences.

But if researchers must consider explicitly the extent that the work they plan to do is ethical, they may find that they cannot conduct the experiment in the way they wish. For example, experimentalists may feel that the only way to manipulate a required variable is to use deception. However, they may find that a particular deception could be ethically problematic and, as a result, to be ethical the design may need to be much less clear-cut in terms of causal inference because the relevant causal variable is not directly manipulated. Similarly, a researcher may face a dilemma between wanting to have more subjects in an experiment so as to make it easier to find statistically significant results and the ethical desire to minimize intervention and the size of possible harms.

Furthermore, sometimes a researcher cannot fully perceive the ethical concerns that may arise in an experiment and only once in the field recognize that his or her plans should be altered. Given the emphasis on following through with well thought-out designs, changing an experiment while in progress may be seen as invalidating the research. ⁵ Choosing to be ethical in experimentation is difficult, but exceedingly so when researchers see their careers as dependent on the publication of exciting and new results.

It is our view that if journals ignore the ethical concerns in the research submitted for possible publication, then they indirectly encourage researchers to take chances and ignore these concerns as well. Hence, we take seriously our role as monitoring the extent that ethical issues are dealt with in designs and during the conducting of an experiment. Our principal mechanism of enforcing ethical standards is to rely on the Institutional Review Boards and Human Subjects Committees at the researchers’ home institutions. Researchers are expected to receive approval from their respective overseers before conducting experimental research, regardless of the location of that research. In the United States, although there is significant variation across institutions in how the standards are implemented, there are in general some common standards concerning deception, informed consent, vulnerable populations, etc. ⁶ A number of other countries such as Canada and the United Kingdom have similar procedures. Hence, we have as a requirement before publication of any new experimental work that the authors submit a copy of the approval from their respective institutional body or – if such approval is not available – a statement as to why it is not available. ⁷

That said, we recognize as well that people define ethical research in a variety of ways and that there are legitimate disagreements as to how do so. Moreover, most institutions outside the U.S. do not have active review boards or committees for social science experiments. So requiring approval is just a first step. The next step is to carefully consider when we receive a paper whether the experiment reported is ethical, and, if necessary, to press the authors to address these issues prior to acceptance. We expect this consideration of ourselves, our Associate Editors, and our reviewers.

We have adopted as our foundation the standards prevalent in the United States as represented in the Federal Policy for Protection of Human Subjects, the so-called Common Rule. ⁸ The Common Rule provides a benefit/cost calculation to determine whether research is ethical and applies a number of guidelines and requirements that are designed to minimize costs. Although the Common Rule is far from perfect, we see it as the best available basis from which to approach ethical issues. From this perspective, given that estimating the benefits from political science experiments is at best a random draw (since if we knew the benefits we would gain from the experiment we would not need to conduct it!), the key issue in experimentation in political science is whether the experiment results in subjects experiencing more than minimal risk in comparison to a daily life standard. ⁹

So far, gladly, we have not confronted a situation where we felt that a paper had such serious ethical issues such that we would not publish it for that reason alone. But we hope that our view that journals do have a role in monitoring the ethical standards used by researchers in the discipline will not be limited to ourselves. This reality is especially so in the current political climate. We believe that – as a discipline – if we are not able to do a good job monitoring ourselves – as well as convincing others that we are doing so – then certainly others will be more than happy to step in and do so for us. Given the time and effort that so many in the discipline have put into thinking about experimental research, we believe that political scientists are the ones best qualified to figure out how to appropriately design ethics-based standards while still preserving the ability to conduct valuable and interesting research. We are also confident that if political scientists do not embrace this task, we will find these decisions taken away from us. So there is an imperative to address these sorts of questions sooner as opposed to later.

We have laid out how we are doing so at JEPS in the preceding pages, and we hope other editors will join us in the coming months. We also are happy to receive suggestions from our readers as to how we can improve our policies moving forward. For there are still undoubtedly many thorny questions which we currently ignore but we should address or will arise in the future. With this in mind, we will host a panel at MPSA and (hopefully, if accepted) APSA on the subject of “Informed Consent in Field Experiments”. We do so in an effort to start a dialogue on an extremely difficult topic, but one that fits squarely in the category of subjects that we as a discipline should be discussing. We hope that by getting enough people in a room to think through this issue together, we can acquire some good ideas for moving forward, and we look forward to featuring some of that discussion in the pages of JEPS in the future.