Chapter 5 delves into the steps that occur prior to, during, and after an experiment – including arriving at questions to explore with an experiment; documenting the steps in the process of conducting an experiment; and considering whether to replicate one’s findings after an experiment. This discussion touches on the themes of the aforementioned open science movement, offering in many instances a cautionary perspective.

In recent years, the credibility of social science has been tarnished by widely discussed replication failures and a lack of reporting about what exactly researchers did when conducting their studies.In response, scholars, policymakers, and the public have called for greater transparency in social science research.In this chapter, I emphasize that transparency is an important public good.However, because individual researchers lack incentives to contribute to this public good, institutional solutions are needed.I discuss three institutions that facilitate transparency in experimental research:1) pre-registration, 2) reporting guidelines, and 3) the Data Access and Research Transparency (DA-RT) initiative.I also offer recommendations for what kinds of information researchers should pre-register and report in their published articles and appendices.I conclude with a discussion of how researchers might be incentivized to make greater use of these institutions when designing, conducting, and publishing their experiments.

Publication bias and p-hacking are threats to the scientific credibility of experiments. If positive results are more likely to be published than null results conditional on the quality of the study design, then effect sizes in meta-analyses will be inflated and false positives will be more likely. Publication bias also has other corrosive effects as it creates incentives to engage in questionable research practices such as p-hacking. How can these issues be addressed such that the credibility of experiments is improved in political science? This chapter discusses seven specific solutions, which can be enforced by both formal institutions and informal norms.

New methods and tools have emerged over the past decade to address pervasive problems of publication bias, p-hacking, and lack of reproducibility. This chapter reviews some of these advances, considering the strengths and shortcomings of each. Meta-analysis, study registration, pre-analysis plans, improved disclosure policies, and open data are all considered.