This paper examines the replicability of studies in design research triggered by the replication crisis in psychology. It highlights the importance of replicating studies to ensure the robustness of research results and examines whether the description in a publication is sufficient to replicate. Therefore, the publication of a reference study was analysed and a replication study was conducted. The design of the replication study appears similar to the reference study, but the results differ. Possible reasons for the differences and implications for replication studies are discussed.

Instrumental variable (IV) strategies are widely used in political science to establish causal relationships, but the identifying assumptions required by an IV design are demanding, and assessing their validity remains challenging. In this paper, we replicate 67 articles published in three top political science journals from 2010 to 2022 and identify several concerning patterns. First, researchers often overestimate the strength of their instruments due to non-i.i.d. error structures such as clustering. Second, IV estimates are often highly uncertain, and the commonly used t-test for two-stage-least-squares (2SLS) estimates frequently underestimate the uncertainties. Third, in most replicated studies, 2SLS estimates are significantly larger in magnitude than ordinary-least-squares estimates, and their absolute ratio is inversely related to the strength of the instrument in observational studies—a pattern not observed in experimental ones—suggesting potential violations of unconfoundedness or the exclusion restriction in the former. We provide a checklist and software to help researchers avoid these pitfalls and improve their practice.

It is a long known problem that the preferential publication of statistically significant results (publication bias) may lead to incorrect estimates of the true effects being investigated. Even though other research areas (e.g., medicine, biology) are aware of the problem, and have identified strong publication biases, researchers in judgment and decision making (JDM) largely ignore it. We reanalyzed two current meta-analyses in this area. Both showed evidence of publication biases that may have led to a substantial overestimation of the true effects they investigated. A review of additional JDM meta-analyses shows that most meta-analyses conducted no or insufficient analyses of publication bias. However, given our results and the rareness of non-significant effects in the literature, we suspect that biases occur quite often. These findings suggest that (a) conclusions based on meta-analyses without reported tests of publication bias should be interpreted with caution and (b) publication policies and standard research practices should be revised to overcome the problem.

Are difficult decisions best made after a momentary diversion of thought? Previous research addressing this important question has yielded dozens of experiments in which participants were asked to choose the best of several options (e.g., cars or apartments) either after conscious deliberation, or after a momentary diversion of thought induced by an unrelated task. The results of these studies were mixed. Some found that participants who had first performed the unrelated task were more likely to choose the best option, whereas others found no evidence for this so-called unconscious thought advantage (UTA). The current study examined two accounts of this inconsistency in previous findings. According to the reliability account, the UTA does not exist and previous reports of this effect concern nothing but spurious effects obtained with an unreliable paradigm. In contrast, the moderator account proposes that the UTA is a real effect that occurs only when certain conditions are met in the choice task. To test these accounts, we conducted a meta-analysis and a large-scale replication study (N = 399) that met the conditions deemed optimal for replicating the UTA. Consistent with the reliability account, the large-scale replication study yielded no evidence for the UTA, and the meta-analysis showed that previous reports of the UTA were confined to underpowered studies that used relatively small sample sizes. Furthermore, the results of the large-scale study also dispelled the recent suggestion that the UTA might be gender-specific. Accordingly, we conclude that there exists no reliable support for the claim that a momentary diversion of thought leads to better decision making than a period of deliberation.

A critical barrier to generating cumulative knowledge in political science and related disciplines is the inability of researchers to observe the results from the full set of research designs that scholars have conceptualized, implemented, and analyzed. For a variety of reasons, studies that produce null findings are especially likely to be unobserved, creating biases in publicly accessible research. While several approaches have been suggested to overcome this problem, none have yet proven adequate. We call for the establishment of a new discipline-wide norm in which scholars post short “null results reports” online that summarize their research designs, findings, and interpretations. To address the inevitable incentive problems that earlier proposals for reform were unable to overcome, we argue that decentralized research communities can spur the broader disciplinary norm change that would bring advantage to scientific advance. To facilitate our contribution, we offer a template for these reports that incorporates evaluation of the possible explanations for the null findings, including statistical power, measurement strategy, implementation issues, spillover/contamination, and flaws in theoretical priors. We illustrate the template’s utility with two experimental studies focused on the naturalization of immigrants in the United States and attitudes toward Syrian refugees in Jordan.

One of the strongest findings across the sciences is that publication bias occurs. Of particular note is a “file drawer bias” where statistically significant results are privileged over nonsignificant results. Recognition of this bias, along with increased calls for “open science,” has led to an emphasis on replication studies. Yet, few have explored publication bias and its consequences in replication studies. We offer a model of the publication process involving an initial study and a replication. We use the model to describe three types of publication biases: (1) file drawer bias, (2) a “repeat study” bias against the publication of replication studies, and (3) a “gotcha bias” where replication results that run contrary to a prior study are more likely to be published. We estimate the model’s parameters with a vignette experiment conducted with political science professors teaching at Ph.D. granting institutions in the United States. We find evidence of all three types of bias, although those explicitly involving replication studies are notably smaller. This bodes well for the replication movement. That said, the aggregation of all of the biases increases the number of false positives in a literature. We conclude by discussing a path for future work on publication biases.

Federal agencies invest taxpayer dollars every year in conservation programs that are focused on improving a suite of ecosystem services produced on private lands. A better understanding of the public benefits generated by federal conservation programs could help improve governmental efficiency and economic welfare by providing science-based evidence for use in policy decision-making regarding targeting of federal conservation investments. Of specific concern here are conservation investments made by the U.S. Department of Agriculture (USDA). While previous research has shown that efficiency gains are possible using cost-benefit analysis for targeting conservation investments, agency-wide implementation of this approach by policy makers has been constrained by the limited availability of location-specific information regarding conservation benefits. Cost-effective opportunities for integrating location-specific ecosystem service valuation research with USDA conservation decision-making include: (1) institutionalizing funding of comparable studies suitable for benefit transfer, (2) utilizing non-traditional data sources for research complementing benefit transfer, and (3) creating a state-of-the-art program for developing and communicating research in ecosystem service valuation exemplifying the highest standards of scientific conduct.

Systematic reviews in mental health have become useful tools for health professionals in view of the massive amount and heterogeneous nature of biomedical information available today. In order to determine the risk of bias in the studies evaluated and to avoid bias in generalizing conclusions from the reviews it is therefore important to use a very strict methodology in systematic reviews. One bias which may affect the generalization of results is publication bias, which is determined by the nature and direction of the study results. To control or minimize this type of bias, the authors of systematic reviews undertake comprehensive searches of medical databases and expand on the findings, often undertaking searches of grey literature (material which is not formally published). This paper attempts to show the consequences (and risk) of generalizing the implications of grey literature in the control of publication bias, as was proposed in a recent systematic work. By repeating the analyses for the same outcome from three different systematic reviews that included both published and grey literature our results showed that confusion between grey literature and publication bias may affect the results of a concrete meta-analysis.

Introduction: Non-publication of trial findings results in research waste and compromises medical evidence and the safety of interventions in child health. The objectives of this study were to replicate, compare and contrast findings of a previous study (Klassen et al., 2002) to determine the impact of ethical and editorial mandates to register and publish findings. Methods: Abstracts accepted to the Pediatric Academic Societies meetings (2008-2011) were screened in duplicate to identify Phase-III RCTs enrolling pediatric populations. Subsequent publication was ascertained through a search of electronic databases. Study internal validity was measured using Cochrane Risk of Bias and Jadad Scale, and key variables (e.g., trial design, study stage) were extracted. Pearson X2, t-tests and Wilcoxon rank sum tests were used to examine association between variables and publication status. Logistic regression, log-rank tests, rank correlation and Egger regression were used to assess predictors of publication, time to publication and publication bias, respectively. Results: Compared to our previous study, fewer studies remained unpublished (27.9% vs 40.9%, p=.007). Abstracts with higher sample sizes (p=0.01) and those registered in ClinicalTrials.gov were more likely to be published (p<.0001). There were no differences in quality measures/risk of bias or in preference for positive results (p=0.36) between published and unpublished studies. Mean time to publication was 26.5 months and published manuscripts appeared most frequently in Pediatrics, the Journal of Pediatrics, and Pediatric Emergency Care. The funnel plot (p=0.04) suggests a reduced but ongoing existence of publication bias among published studies. Overall, we observed a reduction in publication bias and in preference for positive findings, and an increase in study size and publication rates over time. Conclusion: Despite heightened safeguards and editorial policy changes in recent decades, publication bias remains commonplace and presents a threat to assessing the efficacy and effectiveness of interventions in child health. Our results suggest a promising trend towards a reduction in publication bias over time and positive impacts of trial registration. Further efforts are needed to ensure the entirety of evidence can be accessed when assessing treatment effectiveness.