Protecting the Community: Lessons from the Montana Flyer Project

Abstract

In October 2014, flyers appeared in mailboxes of Montana voters that positioned nominees for the state Supreme Court according to an ideological scale. The study, authored by researchers from Stanford and Dartmouth, was met with public outrage. The Commissioner of Political Practices in Montana asked me to vet the ethics of the study. The investigation led me to conclude that current Institutional Review Board (IRB) practices are inadequate for evaluating research involving field experiments in political science because there is no explicit attention in the process to protect a community. I believe the IRB should mandate that researchers explicitly address implications about how their research could affect the communities they study.

In October 2014, flyers appeared in the mailboxes of Montanans that plotted where candidates for two nonpartisan state Supreme Court seats aligned ideologically compared to President Obama and Republican presidential nominee Mitt Romney. The flyers also included images of the Great Seal of Montana; the charge “Take this to the polls!”; and the statement, “This guide was created as part of a joint research project at Stanford and Dartmouth.” This unexpected campaign development caused confusion, including about how researchers could ethically inject themselves into the middle of an election (Johnson 2015a; McCulloch v. Stanford and Dartmouth 2015).

I was uniquely situated as an observer. I received the flyer in the mail and fielded numerous queries in Montana about why researchers from Stanford and Dartmouth would care about judicial elections in Montana. In February 2015, the Montana Commissioner of Political Practices (COPP), Jonathan Motl, recruited me to vet the ethics of the study. I had access to the investigative file that included, among other documents, written responses submitted by Stanford and Dartmouth to inquiries from the COPP; the IRB application from one of the researchers to the Dartmouth Committee for the Protection of Human Subjects for a prototype study in New Hampshire; and e-mails written by the researchers. The focus of this article is to suggest that the controversies surrounding the Montana flyer project demonstrate that standard IRB processes are not an adequate safeguard for protecting the broader community.

Political science field experiments reflect ethical challenges that current IRB practices, developed to protect individual subjects in research fields such as biomedicine, are not suited to review (Humphreys 2014). IRBs have chosen to adopt a narrow interpretation of the ethical principles of “respect, benefice, and justice” stated in the Belmont Report published in 1979 for the study of human subjects (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research 1979). The review process focuses on protection for individuals but generally excludes questions involving people in the aggregate (Desposato 2016; Teele 2014). The limitation may be reasonable for research conducted in the laboratory sciences; however, a political science field experiment about elections involves a different set of ethical concerns (Humphreys 2014).

Some warnings articulated by ethics scholars about field research in political science were prescient in forecasting the problems evident in the Montana case: the failure to consider how a study will affect the community (Teele 2014); the lack of informed consent for subjects (Zimmerman 2016); the lack of review in the locality where the research is conducted (Desposato 2016; Zeichmeister 2016); the interpretation of “benefice” only in terms of risk and benefits for individuals (Gubler and Selway 2016; Zimmerman 2016); and the possibility that a study could influence the outcome of an election (Zimmerman 2016).¹ Another layer of trouble for the researchers involved Montana election law.

Some warnings articulated by ethics scholars about field research in political science were prescient in forecasting the problems evident in the Montana case: the failure to consider how a study will affect the community (Teele 2014); the lack of informed consent for subjects (Zimmerman 2016); the lack of review in the locality where the research is conducted (Desposato 2016; Zeichmeister 2016); the interpretation of “benefice” only in terms of risk and benefits for individuals (Gubler and Selway 2016; Zimmerman 2016); and the possibility that a study could influence the outcome of an election (Zimmerman 2016).

THE IRB REVIEW

In May 2014, the Dartmouth researcher submitted an application titled “Information and Extremism in US Primary Elections” to the Dartmouth Committee for the Protection of Human Subjects. The study proposed sending voter guides to selected residents of the New Hampshire first congressional district for the upcoming primary election for “Governor, US House, US Senate, and other races” to determine whether the guides had “any impact on voter turnout, specifically among moderate, independent, or unaffiliated voters, and whether the guide affects vote choice or ballot rolloff.” The researcher informed the Dartmouth IRB that “hundreds of academics and political officials have conducted similar studies across the nation” and indicated that “we do not plan on informing respondents that they are participants in an academic, educational study. We view the study as having minimal risk to the participants” (Investigative File 2015, Dartmouth IRB application).

Several mistakes were made during the approval process. The researchers failed to follow protocols because only the Dartmouth—and not the Stanford—IRB reviewed the New Hampshire study. Moreover, the researchers failed to submit the Montana study (and a study in the 52nd congressional district in California) to any IRB. A new subject population needs approval prior to an extension of treatment. Furthermore, the Montana study—in contrast to New Hampshire—targeted voters at the ideological poles rather than moderates; involved flyers for only two judicial races; and was a study that targeted the general election and not a primary ballot (Johnson 2015a; McCulloch v. Stanford and Dartmouth 2015).

The Dartmouth IRB also erred by giving insufficient attention to detail during the approval process. The director of the Committee for the Protection of Human Subjects signed a letter designating the project exempt from further review but not under the regulatory “category 2” requested by the researcher. The category 2 exemption includes “survey procedures, interview procedures, or observation of public behavior,” which is the correct designation for the flyer project because there was no category that precisely included field experiments. The Dartmouth IRB, without explanation, granted a “category 4” exemption, which is a regulatory category for the “collection of existing data, documents, records, pathological or diagnostic specimens in such a manner that subjects cannot be identified” (Investigative File 2015, Dartmouth IRB approval). There is no record of any follow-up from the researcher despite the mistake made by the IRB in granting the exemption.

The director of the Dartmouth Committee for the Protection of Human Subjects approved the flyer study on August 28, 2014, for the New Hampshire primary scheduled on September 9, 2014 (Investigative File 2015, Dartmouth IRB approval). No explanation was provided by Stanford or Dartmouth for why neither the California nor the Montana study was submitted for IRB review—although time constraints were an obvious challenge for the researchers.

THE POLITICAL ENVIRONMENT IN MONTANA

Conditions were ripe in Montana for a negative response to any study involving flyers. Beginning in 2008, flyers targeting candidates and financed by dark-money groups began appearing in the mailboxes of Montanans in what were once low-cost primary elections for seats in the legislature and the Supreme Court. By 2014, six years of news about dark money, attack flyers, legal challenges, and outside spending inundating Montana left much of the electorate dismayed (Blumenthal 2016; Davidson 2012).

A nonpartisan Supreme Court primary occurred in June 2014 with unremarkable results. The two incumbent Supreme Court justices, Jim Rice and Mike Wheat, won by lopsided margins against two less-well-known opponents, W. David Herbert and Lawrence Van Dyke, respectively. The same four candidates appeared again on the November ballot; however, the character of the race for the seat held by Wheat had changed. Outside conservative groups poured $1.36 million into Montana to bolster the candidacy of Van Dyke (Johnson 2015a).

THE FLYERS

Stanford and Dartmouth explained that the researchers chose the Montana Supreme Court races “primarily because they were immediately preceded by a June primary election which the exact same candidates had run.” About 25% of the primary voters left both judicial contests blank. This was useful as a baseline and “neither judicial race had been closely contested in the primary.... [i]n the context of previous Montana judicial elections, the researchers determined that the research study as designed would not change the outcome of either contest” (Investigative File 2015, Stanford response to Q. 22). The COPP concluded that the researchers were “naïve” to anticipate that no change in election results would make the study acceptable (McCulloch v. Stanford and Dartmouth 2015).

The COPP concluded that the researchers were “naïve” to anticipate that no change in election results would make the study acceptable (McCulloch v. Stanford and Dartmouth 2015).

Between October 4 and October 16, the researchers communicated via e-mail with an outside consultant hired to facilitate mailing the flyers and a graphic designer contracted to develop them. Although the researchers never raised questions about legal issues in their e-mail threads, the outside consultant voiced concerns. On October 11, she wrote to one researcher and asked, “Are you OK mailing without a disclaimer again? I just want to confirm that.” The researcher did not respond to the query via e-mail, although communication off e-mail may have occurred. The consultant again raised the question about legalities via e-mail to all three researchers on October 15, when she made five logistical inquiries about mailing the flyers. Her third question was: “I know we talked about the disclaimer issue before. I have no knowledge of what is legally req’d in CA or MT so I am going to assume you vetted these through someone who does know what’s required in these states, and the language you have on the artwork is sufficient?” One researcher replied minutes later, answering all of the logistical questions except the third question, which he skipped. None of the researchers ever directly responded. However, later on October 15, a researcher wrote that he had “one final request,” which was the disclaimer “Paid for by researchers at Stanford University and Dartmouth College” with a Stanford mailing address included on the back of the flyers (Investigative File 2015, researcher e-mails). Stanford and Dartmouth explained the researchers’ silence about legal vetting in response to the COPP’s inquiries as follows: the researchers neither “sought or received advice from legal counsel during the course of this research project” and did not consider Montana’s Campaign Practice Act (Investigative File 2015, Stanford response to Q. 18 and Q. 19).

Approximately 15% of the population of Montana served as the study’s treatment group and received flyers: 64,265 flyers to voters identified as moderate to liberal leaning in Democratic precincts and 38,515 flyers to voters identified as moderate to conservative leaning in Republican precincts. The researchers hoped for ideological balance and assumed that fewer Democrats would vote in a midterm election; thus, more flyers were mailed to Democratic leaners (Investigative File 2015, Stanford response to Q. 23). The stated purpose of the study was to evaluate whether additional ideological information about candidates would lead to less voter rolloff than in precincts that did not receive the flyers (Investigative File 2015, Stanford response to Q. 4). The flyers also now included the image of the Great Seal, which was added by the graphic designer at the behest of a researcher without any consideration of potential legal implications (Investigative File 2015, Stanford response to Q. 8).

THE RECEPTION IN MONTANA

The flyers began arriving in mailboxes on October 22 and informal complaints immediately began flooding the COPP’s office. Motl wrote in the decision, “Judging from the number of complaints received by the COPP staff, Montanans intuitively thought the Montana flyer project was a flawed piece of election activity” (McCulloch v. Stanford and Dartmouth 2015). One concern was that the research project was designed to affect vote tallies. A US senator from Montana, Jon Tester, summarized these complaints by describing that the intent of the Stanford–Dartmouth flyer was “voter manipulation” and he demanded further investigation (Dennison 2014). Some suspected that the project was financed by dark-money groups, which prompted the COPP to inquire: “[w]as any funding received directly or indirectly from the Koch brothers?” Both institutions answered “no” (Investigative File 2015, Stanford response to Q. 3).

Other Montanans, as noted by Motl in his decision, “saw the flyer as wrongfully appropriating use of the Great Seal of the State of Montana and wrongfully campaigning without reporting or disclosure”; they were incredulous that such a project could have received approval. He mentioned that a “medical doctor went so far as to independently review the published Dartmouth IRB procedures, concluding that the Montana flyer project could not have received legitimate IRB approval” (McCulloch v. Stanford and Dartmouth 2015). The flyers placed Wheat near Obama on the ideological continuum, underscoring arguments promoted by independent conservative groups about Wheat being an Obama “shill”—a challenge for a candidate in a state where the president’s approval rating was around 30% throughout 2014 (Gallup 2015). Therefore, Democratic-leaning voters were suspicious of the motivation for the study, and they were most likely to receive the flyers in the mail.

On October 24, a formal complaint was filed by Linda McCulloch, the Montana secretary of state. At Motl’s request, an apology letter signed by the presidents of Stanford and Dartmouth was sent to Montanan households that received the flyer, stating that “no research study should risk an election” and requested voters to ignore the flyers (McCulloch v. Stanford and Dartmouth 2015). Despite the apology letter, in its response to the questions posed by the COPP, Stanford wrote: “[t]he use of the seal was a mistake and the primary reason why Stanford agreed with your office to mail the apology letter.” Stanford made no mention in the response to the COPP of disrupting the election because of the study design, which was intended to influence vote tallies (Investigative File 2015, Stanford response to Q. 5).

THE DECISION

In May 2015, Motl found that the flyers were a form of “express” rather than “issue” advocacy and therefore were not in compliance with the Montana Campaign Practices Act (McCulloch v. Stanford and Dartmouth 2015). A spokesperson for Stanford University defended the study by also contradicting the apology made by the president of Stanford to voters in Montana. She said, “The Stanford/Dartmouth mailing at issue was a protected First Amendment university research project about voter turnout....Stanford disagrees with the finding of the commissioner” (Richman 2015). This statement, reported in the Mercury News of San Jose, was undercut in the same article by a former California campaign regulator, Dan Schnur. He stated that Motl’s ruling looked “solid” and thought the study “incredibly condescending. [The researchers] would never dream about sending out this mailer to voters in Palo Alto or Atherton.... But they assumed these were dumb Montanans who wouldn’t know any better” (Richman 2015). Schnur was mistaken on this point. The researchers had conducted studies in New Hampshire and California as well as Montana; they had not considered how voters anywhere would react to the flyers.

THE SETTLEMENT

In March 2017, Stanford and Dartmouth resolved all aspects of the complaint by paying a total of $13,599.58, with each school sending a check for $6,799.79 to the state of Montana (Settlement 2017). The settlement referenced how Stanford and Dartmouth had cooperated with the investigation, including mailing an apology—at a cost of $50,000—to the Montanans who received the flyers. Furthermore, “each of the Respondents represents that it has taken additional positive internal steps to ensure that future academic election-related projects are conducted in a manner to avoid regulatory issues.” However, the exact nature of these changes was not specified. Motl recognized these measures as “actions that trigger mitigation of the social debt assessed by this type of formal settlement agreement.” Nonetheless, as noted in the settlement, Stanford and Dartmouth continued to “deny any violation of Montana law” and “disagree with the conclusions of law” made by the COPP; they agreed to make the payment only to “avoid litigation” (Settlement 2017). The cooperation of Stanford and Dartmouth during the course of the investigation resulted in a relatively small financial penalty.

Questions include: What is the role of the IRB? What is the role of self-regulation on the part of academics? Should political scientists conduct field research involving projects that intend to influence vote tallies?

RESEARCH ETHICS

There are various perspectives on how to assess what went awry in Montana and how to prevent recurrences. Questions include: What is the role of the IRB? What is the role of self-regulation on the part of academics? Should political scientists conduct field research involving projects that intend to influence vote tallies? The existing procedure for IRB approval may unintentionally foster a narrow mindset compartmentalizing ethical considerations by focusing safeguards solely on the individual subject (Johnson 2015b). Many researchers are not mindful of the issue raised by Michelson (2016) that “an approved project is not always an ethical project.” Indeed, Schrag (2015) argued that the IRB may not have jurisdiction over the Montana flyer study because the data gathered was not done so unambiguously through “intervention or interaction with the individual”—a position rejected by Stanford, Dartmouth, and the COPP. The fact that Schrag can plausibly make that argument demonstrates the inherent limitations in the existing IRB process.

Furthermore, even if the researchers had followed IRB protocol at Dartmouth, the study very well could have received approval unchanged and likewise at other institutions. The underlying reason is that concerns about the community are not even an implicit consideration. From that vantage point, as noted by Desposato (2014), nothing worse than a “paper cut” will result. Potential safeguards that could mitigate these problems—such as provisions in the Common rule that encourage researchers to inform their subjects of their results—are generally not followed (Desposato 2016). A lesson from the Montana flyer project is that academics must be cognizant of the limitations of the IRB process and seek out legal advice when appropriate.

Finally, what are the ethical implications for field experiments involving elections? If the Stanford–Dartmouth flyers had implored recipients to vote as a civic duty and avoided mentioning candidates, the flyers would have passed muster with the Montana Campaign Practice Act. Of course, any field-experiment study involving mailings designed to affect voting in some way will influence vote tallies. There is potential that affected populations will resent the interference and the inability to give consent and that the outcome of an election could change (Johnson 2015b). The ethics are complex; therefore, researchers and IRBs should be aware of the communities in which experiments will occur.

CONCLUSION

Election flyers remain controversial in Montana. Campaign practices and questions revolving around the integrity of elections remain at the forefront of attention in state legislative and Supreme Court races. No research study involving flyers in Montana is advisable in the current political environment.

Standard IRB practices, with a focus on the individual subject, are not an adequate safeguard for research involving field experiments and elections. Field experiments have become more common in recent years because they are a useful method for gathering data about questions such as what motivates people to vote and which types of appeals are most effective. This trend in research exemplifies why IRB practices need revision. The IRB should include explicit instructions for researchers to consider whether research has the potential to adversely affect a community. The legal controversy and negative public reaction elicited by an academic study indicate that integrating political research into a system established for ethical considerations related to the natural and medical sciences has proven inadequate.

RECOMMENDATION

The mandate of the IRB should be extended in scope beyond the protection of an individual subject. All IRBs should incorporate a question that requires researchers to address how a study could affect the broader community and to consider measures to mitigate potential adverse consequences. Researchers should strive to understand the nature of the community they plan to study. Attention to these matters could have extinguished the Montana study before it ever commenced.