Hostname: page-component-78c5997874-lj6df Total loading time: 0 Render date: 2024-11-13T01:46:02.214Z Has data issue: false hasContentIssue false
  • English
  • Français

How (Not) to Reproduce: Practical Considerations to Improve Research Transparency in Political Science

Published online by Cambridge University Press:  16 September 2021

R. Michael Alvarez Open the ORCID record for R. Michael Alvarez [Opens in a new window]  and
Simon Heuberger
R. Michael Alvarez*
Affiliation:
California Institute of Technology, USA
Simon Heuberger*
Affiliation:
American University, USA
Get access Rights & Permissions [Opens in a new window]

Abstract

In recent years, scholars, journals, and professional organizations in political science have been working to improve research transparency. Although better transparency is a laudable goal, the implementation of standards for reproducibility still leaves much to be desired. This article identifies two practices that political science should adopt to improve research transparency: (1) journals must provide detailed replication guidance and run provided material; and (2) authors must begin their work with replication in mind. We focus on problems that occur when scholars provide research materials to journals for replication, and we outline best practices regarding documentation and code structure for researchers to use.

Type
Article
Information
PS: Political Science & Politics , Volume 55 , Issue 1 , January 2022 , pp. 149 - 154
Copyright
© The Author(s), 2021. Published by Cambridge University Press on behalf of the American Political Science Association

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

REFERENCES

Breznau, Nate. 2021. “I Saw You in the Crowd: Credibility, Reproducibility, and Meta-Utility.” PS: Political Science & Politics 54 (2): 1–5. DOI:10.1017/S1049096520000980.Google Scholar
Bryan, Jenny. 2018. “Ode to the Here Package.” https://github.com/jennybc/here_here.Google Scholar
Clemens, Michael A. 2017. “The Meaning of Failed Replications: A Review and Proposal.” Journal of Economic Surveys 31 (1): 326–42.10.1111/joes.12139CrossRefGoogle Scholar
Colaresi, Michael. 2016. “Preplication, Replication: A Proposal to Efficiently Upgrade Journal Replication Standards.” International Studies Perspectives 17 (4): 367–78.Google Scholar
Coughlin, Steven S. 2017. “Reproducing Epidemiologic Research and Ensuring Transparency.” American Journal of Epidemiology 186 (4): 393–94.10.1093/aje/kwx065CrossRefGoogle ScholarPubMed
Dafoe, Allan. 2014. “Science Deserves Better: The Imperative to Share Complete Replication Files.” PS: Political Science & Politics 47 (1): 60–66.Google Scholar
Engzell, Per, and Rohrer, Julia M.. 2020. “Improving Social Science: Lessons from the Open Science Movement.” PS: Political Science & Politics 54 (2): 1–4.Google Scholar
Eubank, Nicholas. 2016. “Lessons from a Decade of Replications at the Quarterly Journal of Political Science .” PS: Political Science & Politics 49 (2): 273–76.Google Scholar
Freese, Jeremy. 2007. “Replication Standards for Quantitative Social Science: Why Not Sociology?” Sociological Methods & Research 36 (2): 153–72.10.1177/0049124107306659CrossRefGoogle Scholar
Freese, Jeremy, and Peterson, David. 2017. “Replication in Social Science.” Annual Review of Sociology 43 (1): 147–65.CrossRefGoogle Scholar
Gertler, Paul, Galiani, Sebastian, and Romero, Mauricio. 2018. “How to Make Replication the Norm.” Nature 554 (7690): 417–19.10.1038/d41586-018-02108-9CrossRefGoogle ScholarPubMed
Gherghina, Sergiu, and Katsanidou, Alexia. 2013. “Data Availability in Political Science Journals.” European Political Science 12:333–49.CrossRefGoogle Scholar
Gleditsch, Nils Petter, Metelits, Claire, and Strand, Håvard. 2003. “Posting Your Data: Will You Be Scooped or Will You Be Famous?” International Studies Perspectives 4 (1): 9–95.Google Scholar
Harden, Jeffrey J., Sokhey, Anand E., and Wilson, Hannah. 2018. “Replications in Context: A Framework for Evaluating New Methods in Quantitative Political Science.” Political Analysis 27:119–25.CrossRefGoogle Scholar
Heuberger, Simon. 2018. “Insufficiencies in Data Material: A Replication Analysis of Muchlinski, Siroky, He, and Kocher (2016).” Political Analysis 27:114–18.CrossRefGoogle Scholar
Ishiyama, John. 2014. “Research Transparency, and Journal Publications: Individualism, Community Models, and the Future of Replication Studies.” PS: Political Science & Politics 47 (1): 78–83.Google Scholar
Janz, Nicole. 2016. “Bringing the Gold Standard into the Classroom: Replication in University Teaching.” International Studies Perspectives 17:392–407.Google Scholar
Janz, Nicole, and Freese, Jeremy. 2020. “Replicate Others as You Would Like to Be Replicated Yourself.” PS: Political Science & Politics 54 (2): 1–4.Google Scholar
Kapiszewski, Diana, and Karcher, Sebastian. 2020. “Transparency in Practice in Qualitative Research.” PS: Political Science & Politics 54 (2): 1–7.Google Scholar
King, Gary. 1995. “Replication, Replication.” PS: Political Science & Politics 28 (3): 444–52.Google Scholar
Lall, Ranjit. 2016. “How Multiple Imputation Makes a Difference.” Political Analysis 24 (4): 414–33.CrossRefGoogle Scholar
Lupia, Arthur. 2020. “Practical and Ethical Reasons for Pursuing a More Open Science.” PS: Political Science & Politics 54 (2): 1–4.Google Scholar
Lupia, Arthur, and Elman, Colin. 2014. “Openness in Political Science: Data Access and Research Transparency.” PS: Political Science & Politics 47 (1): 19–42.Google Scholar
Miguel, Edward, Camerer, Colin, Casey, Katherine, Cohen, Jason, Esterling, Kevin, Gerber, Alexander, et al. 2014. “Promoting Transparency in Social Science Research.” Science 343:30–31.10.1126/science.1245317CrossRefGoogle ScholarPubMed
Muchlinski, David A., Siroky, David, He, Jingrui, and Kocher, Matthew A.. 2018. “Seeing the Forest Through the Trees.” Political Analysis 27 (1): 111–13.CrossRefGoogle Scholar
Neunhoeffer, Marcel, and Sternberg, Sebastian. 2018. “How Cross-Validation Can Go Wrong and What to Do About It.” Political Analysis 27 (1): 101–106.CrossRefGoogle Scholar
Nosek, Brian A., Alter, George, Banks, George C., Borsboom, Denny, Bowman, Sara D., Breckler, Steven J., et al. 2015. “Promoting an Open Research Culture.” Science 348 (6242): 1422–25. DOI: 10.1126/science.aab2374.10.1126/science.aab2374CrossRefGoogle ScholarPubMed
Open Science Collaboration. 2012. “An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science.” Perspectives on Psychological Science 7 (6): 657–60.CrossRefGoogle Scholar
Plesser, Hans E. 2018. “Reproducibility vs. Replicability: A Brief History of a Confused Terminology.” Frontiers in Neuroinformatics 11:76.CrossRefGoogle ScholarPubMed
Rinke, Eike M., and Wuttke, Alexander. 2021. “Open Minds, Open Methods: Transparency and Inclusion In Pursuit of Better Scholarship.” PS: Political Science & Politics 54 (2): 1–4.Google Scholar
Rohlfing, Ingo, Königshofen, Lea, Krenzer, Susanne, Schwalbach, Jan, and Ayjeren Bekmuratovna, R. 2020. “A Reproduction Analysis of 106 Articles Using Qualitative Comparative Analysis, 2016–2018.” PS: Political Science & Politics 54 (2): 1–5.Google Scholar
Shepherd, Bryan E., Peratikos, Meridith B., Rebeiro, Peter F., Duda, Stephany N., and McGowan, Catherine C.. 2017. “A Pragmatic Approach for Reproducible Research with Sensitive Data.” American Journal of Epidemiology 186 (4): 387–92.CrossRefGoogle ScholarPubMed
Wang, Yu. 2018. “Comparing Random Forest with Logistic Regression for Predicting Class-Imbalanced Civil War Onset Data: A Comment.” Political Analysis 27 (1): 107–10.CrossRefGoogle Scholar
Supplementary material: PDF

Alvarez and Heuberger supplementary material

Alvarez and Heuberger supplementary material

Download Alvarez and Heuberger supplementary material(PDF)
PDF 382.6 KB