Open Minds, Open Methods: Transparency and Inclusion in Pursuit of Better Scholarship

The idea of “open scholarship” has enjoyed a remarkable career. From discussions around the production, availability, and analysis of data (Moravcsik 2014) to persistent publication biases (Franco, Malhotra, and Simonovits 2014) and the accessibility of research publications (Bull 2016), “openness” has emerged as a unifying concept in discourses about the discipline (Elman, Kapiszewski, and Lupia 2018; Lupia and Elman 2014; Wuttke 2019).

Despite its ubiquity, the meaning of “open science” or “open scholarship” (OS) ¹ has been ambiguous and contested to the point where scholars have identified different “schools of thought” (Fecher and Friesike 2014). Our goal in this symposium is to advance an integrated concept of OS. We argue that as an academic reform movement aimed at raising the credibility of political research, the idea of openness is best conceived of as covering two orthogonal dimensions: openness as transparency and openness as inclusion. This conception of OS dovetails with philosophical accounts that locate science’s unique epistemic quality in its social character (e.g., Longino 1990; Oreskes 2019; Solomon 2001). The ineluctably social processes of mutual vetting, cumulative learning, and free exchange of views can only enhance the credibility of our work if we open up the research process and open scholarly discussions to diverse participants. These two components of OS reinforce one another in the credible accumulation of knowledge about particular questions. Thus, a key benefit of OS is that it encourages us, the scientific community, to assess whether we are realizing our epistemic potential by posing a simple question: How far open are the doors to our research and did we invite others in?

Thus, a key benefit of OS is that it encourages us, the scientific community, to assess whether we are realizing our epistemic potential by posing a simple question: How far open are the doors to our research and did we invite others in?

So, what exactly does openness imply for scholarship and what does the OS movement aim to achieve? To understand its purposes, it helps to revisit a foundational question about the nature of research: Why do we consider scholarly knowledge claims to be credible or at least more credible than knowledge claims from other sources?

An important tradition in the philosophy of social sciences, going back to at least Merton (1973) and taken up recently by Oreskes (2019), argues that science’s epistemic value lies in its social character. In this view, academic knowledge production is distinctive in that scholars constantly review, scrutinize, and evaluate one another’s work. As scholars, we expose our ideas to critique and engage in a process of cumulative—usually incremental—knowledge creation. In doing so, we should not only “stand on the shoulders of giants” but also examine them like a scrupulous orthopedist. The point of the social process of science is that it uncovers errors and opportunities for improvement. It allows us to achieve more robust insights by making us aware of the triumphs and missteps of others who came before us. Moreover, the social process of scholarly debate, at least in principle, may approximate a consensus that provides credible certainty to interested observers. When, after thorough vetting, a community of scholars agree on a particular scientific knowledge claim, then it no longer represents a private opinion but rather the best available collective expert knowledge on a specific topic—at least for the time being and within that community.

A revelation of the OS movement is that this social process depends on transparency or, more fundamentally, on the observability of our work. Simply stated, the academic research process can be subject to self-corrective action only to the extent that others can see it. Openness as transparency—the first dimension of OS—enables social vetting and mutual learning. The less information we make accessible and the more reclusive we are toward our peers, the lower the chances that our work will be vetted by them in meaningful ways and the dimmer the prospects for useful knowledge production. Hence, the more open we are about our research, the more effectively science’s social mechanisms of error control and cumulative learning can make our knowledge claims more robust.

Expert consensus—conceived broadly as a multiplicity of valid studies coming to similar conclusions—is another social mechanism that can foster the credibility of research, but it requires openness. Under ideal conditions, scientific consensus is a helpful signal of expert agreement. However, when studies with unwelcome findings remain hidden in file drawers, scholarly debate will reflect a distorted subset of the available information. As a result, the epistemic value of scholarly consensus will diminish because it will no longer represent the best available collective knowledge. Therefore, the more we open up access to all meaningful research outcomes, the more reliable and valid the signal emitted by the scientific consensus on a particular question will be.

Another key recognition of the OS movement is the role of openness in terms of access to scholarly debates. Openness as inclusion—the second dimension of OS—is key to realizing the epistemic potential of science because it enriches social vetting and makes scholarly agreement and disagreement more meaningful. Opening up scholarly debates to the greatest reasonably possible diversity of participants minimizes the risk that errors that may have been obvious—depending on one’s lived experiences—are overlooked. Hence, a complete understanding of OS encompasses a commitment to reflecting on the role of race, class, ethnicity, professional status, religion, and other meaningful group boundaries in defining the standards of recognizing academic contributions. In this way, the outlook of OS is similar to that of other academic reform movements working to make science a more egalitarian process in which a wide range of voices is heard, such as Women Also Know Stuff and People of Color Also Know Stuff. Viewing research as an enterprise the epistemic value of which depends on its social character pushes us toward an understanding of OS that implies a principled commitment to making the academic process more socially inclusive.

Openness as inclusion—the second dimension of OS—is key to realizing the epistemic potential of science because it enriches social vetting and makes scholarly agreement and disagreement more meaningful.

An integrated conception of OS that centers science’s social character therefore views both propositions—opening up the research process and opening up scholarly debates—as reflecting the same set of principles. In our view, OS consists of both opening the door to one’s research and inviting others in. Only when we satisfy both conditions can we hope to achieve a process of proper collective scrutiny, understanding, and assessment of our work as well as cumulative insight building from it.

In this perspective, the OS movement is all about interrupting the process of “normal science” and critically interrogating it together: Do our established norms, values, incentives, and rules encourage openness and observability of our work? OS is asking this question in an inclusive exercise of promoting self-critique through openness.

Critics argue that the OS movement falls woefully short of its own objectives. Specifically, they point out that OS itself is lacking in diversity and that it perpetuates steep hierarchical power structures within the discipline, favoring elite institutions in the northern hemisphere that often focus on quantitative methods (Women’s Caucus for Political Science 2017). We think there is an important point to this critique: it is true that many advances associated with the OS movement focused on making number-crunching more transparent. Moreover, it is clear that “doing OS” requires resources, which simply may not be available to some researchers—especially when it comes to acquiring the new skills it requires, from learning how to properly publish reproduction materials to documenting the research process more thoroughly than was common in the past. OS raises the bar of what is expected of us as researchers—but it also works to provide the tools needed to clear it.

Strengthening the academic commons is a key point of OS, focused as it is on providing everyone with the tools and infrastructures needed to conduct transparent and credible research, free of charge and regardless of hierarchical status. If this spirit of open innovation is coupled with a social-inclusiveness perspective, we expect that OS—in the long run—will flatten, not steepen, hierarchies that permeate our scholarly community.

If this spirit of open innovation is coupled with a social-inclusiveness perspective, we expect that OS—in the long run—will flatten, not steepen, hierarchies that permeate our scholarly community.

Likewise, the dual conception of OS as transparent and inclusive casts into doubt the entrenched notions of essentialist barriers between OS and qualitative research traditions. There is no intrinsic contradiction between OS and qualitative thinking. Indeed, interpretivist epistemologies have long been concerned with issues of transparency. Questions of researcher subjectivities in discussions around research reflexivity and positionality (Bourke 2014; Rose 1997) connect seamlessly with the transparency dimension of OS. Additionally, as a continuous exercise of scholarly self-critique, OS necessarily questions academic power relations and, in this sense, aligns well with the analysis of power embedded in critical strains of political research. From this perspective, the principles of OS render it an undertaking that is more diverse in nature than some observers have characterized it.

Based on this broad conception of OS, the objective of this symposium is to extend the discussion of OS to political researchers who have not considered themselves to be active proponents of the movement toward OS. It assembles contributions that, as an ensemble, showcase how we can and should address the two dimensions of OS discussed in this introduction: opening up the research process and our discussions about it. We believe that the measures and advice presented in these articles will increase the epistemic value of political research as an inherently social process. The purpose of this symposium is to challenge the discipline to be transparent and inclusive as we strive to understand the political world around us.

Three articles in the symposium discuss openness as transparency. Kapiszewski and Karcher (2020) present various entry points to transparent research for researchers working in qualitative traditions. The article by Rohlfing et al. (2020) demonstrates how transparency can be evaluated and improved in a context-sensitive manner in specific methodological areas. They review the political research literature using Qualitative Comparative Analysis (QCA), not only documenting a problematic lack of reproducibility but also providing a checklist that QCA researchers can use to improve the transparency and reproducibility of their work. Engzell and Rohrer (2020) offer an interdisciplinary perspective on transparency. Their article discusses what political research can learn from psychology’s “replication crisis” and ensuing “credibility revolution.”

Three more articles address openness as inclusion. Lupia (2020) explains why opening up our work to the scholarly and general public is of paramount importance for trust in social science. Janz and Freese (2020) focus on how the core scientific practice of replication can and should be made more inclusive in our discipline. Arguing that the epistemic ends of OS require us to build a cooperative-including rather than a combative-excluding culture, they illustrate how a culture of “fear of replication” has sidelined replication as a key academic practice. In response, Janz and Freese provide actionable recommendations for how we can recenter replication as an inclusive practice. Finally, Breznau (2020) shows how we can advance political research by opening up to a massively inclusive approach to collaboration, even within the confines of a single project. He shows how a wealth of crowdsourced expertise and perspectives can be brought to bear on research of unique utility that would not have been possible in the traditional “one-to-10 authors” paradigm.