Converting a TESS Acceptance to a JEPS Registered Report

The Journal of Experimental Political Science has helped lead the way in adopting the Registered Report format for journal articles, which combines pre-registration with results-blind review. Time-Sharing Experiments for the Social Sciences offers researchers a unique opportunity to propose an experimental design that undergoes peer review before being fielded on a nationally representative survey. We are pleased to announce a collaborative effort between JEPS and TESS that will expedite the review process at JEPS and allow an accepted TESS proposal to be quickly converted to a Registered Report. This effort will eliminate redundant review processes, thereby reducing the burden on reviewers and shortening the review process for authors. Additionally, by focusing reviewer efforts on the design, rather than the results, this opportunity will reduce the well-known file drawer problem, which has also been documented among TESS studies (Franco, Malhotra, and Simonovits 2014).

How does a Registered Report work?

In a standard review process for a Registered Report, the review process involves two stages. The authors first submit a manuscript prior to collecting data. The manuscript provides details on precisely how the proposed study will be run and analyzed. In Stage 1 of the review process, reviewers focus on the importance of the research question and the soundness of the theory, design, and proposed statistical tests. Once reviewers and editors are satisfied with the design, the manuscript is given an in-principle acceptance. The authors then register the design, conduct the study, and write up the results. They then submit a complete manuscript, which undergoes Stage 2 review. In Stage 2, reviewers focus on whether the design was faithfully carried out and whether the conclusions are warranted, rather than on whether the results are statistically significant. 

How will it work?

Authors who have a TESS proposal accepted for funding should convert their proposal to a manuscript that meets the standards at JEPS for a Registered Report. Crucially, this step takes place prior to data collection by TESS. Authors should submit their manuscript, their final TESS proposal, and a cover letter explaining that they intend to convert their TESS project to a Registered Report and requesting that the editorial team consider their TESS reviews as part of the process. If possible, JEPS editors will treat the TESS reviews as the Stage 1 review and rely on these to make a decision of in-principle acceptance. If the TESS reviews are insufficient to make a decision of in-principle acceptance (see below for more detail), it may not be possible to consider the manuscript as a Registered Report, though it may still undergo an expedited review process. Once a manuscript has received an in-principle acceptance, the authors will be asked to submit their Stage 1 manuscript to a repository (e.g., OSF), which will occur prior to data collection. Once data has been collected and a complete manuscript has been submitted, it will be sent out to the original TESS reviewers (as possible) for a final Stage 2 review that focuses on whether the design was faithfully carried out (rather than on the results).

What should my TESS proposal look like?

In submitting to TESS, authors should follow the established guidelines for proposals by TESS. However, to be considered a Registered Report at JEPS, authors should provide all of the detail that is typically included in a pre-registration (e.g., a methods section, analysis plan, sample size justification, etc.). We recognize that this may not be possible within the confines of a TESS proposal, so authors should provide any additional detail in an appendix and ensure that it is available to TESS reviewers. The clearer these details are to TESS reviewers, the easier it will be to convert the TESS proposal to a Registered Report.

What should my JEPS submission look like?

Authors must submit a Stage 1 manuscript that follows the guidelines for a Registered Report at JEPS. To maximize the chances that the manuscript receives a decision of in-principle acceptance without a further stage of review, authors should minimize any differences in substance or method between the TESS proposal and the JEPS manuscript. Authors should also submit the final TESS proposal and a cover letter, as described above.

What if my TESS submission is not sufficiently detailed to be considered a Registered Report?

If JEPS editors determine that the original TESS submission was not sufficiently detailed to receive an in-principle acceptance, an additional stage of review will need to be conducted before in-principle acceptance. In this case, JEPS editors may need to recontact the TESS reviewers or contact a new reviewer. Due to the timeframe of fielding a TESS study, this process may make it impossible to consider a manuscript as a Registered Report. However, the manuscript may still be considered for expedited review as a Research Article.

When should I submit to JEPS?

As soon as possible after receiving an acceptance from TESS. The review process is time sensitive, as it must take place prior to data collection to be considered a Registered Report. Based on past experience, it may take approximately three months between a TESS acceptance and the study being fielded. Thus, we encourage authors to submit their JEPS manuscript within about one month of the TESS acceptance.

Can I submit a manuscript after data has been collected by TESS?

Yes, but it cannot be considered as a Registered Report. Inclusion of the TESS reviews, however, can expedite the usual review process.

Are TESS Short Studies eligible for pre-acceptance?

No. Short Studies are typically only reviewed internally by TESS PIs. As a result, JEPS would not have a full set of reviews to use for a Stage 1 decision and would not have reviewers to recruit for a Stage 2 decision. 

Is JEPS obligated to publish accepted TESS proposals?

No. However, the original TESS reviews will effectively be treated as a Stage 1 review and thus these reviews will weigh heavily in our decision of whether to grant an in-principle acceptance.

What if my proposal is rejected from TESS?

If a proposal is rejected from TESS and the authors continue to pursue the project, the authors may submit the design as a Registered Report to JEPS, so long as the primary data has not already been collected (in which case it would be considered as a standard submission). Editors will use reviews from TESS, if made available, to expedite the review process as appropriate. Authors can facilitate the process by including a cover letter explaining how they have modified the design in response to prior reviews. The degree to which we can conduct an expedited review process will depend, of course, on the nature of the initial TESS reviews.

If I submit a Registered Report to JEPS am I obligated to publish with JEPS?

Yes.

References

Franco, Annie, Neil Malhotra, and Gabor Simonovits. 2014. “Publication Bias in the Social Sciences: Unlocking the File Drawer.” Science 345(6203): 1502–5.

Process of a Successful Conversion

• Author submits a design to TESS
• TESS accepts the design for fielding after peer review
• Author submits manuscript to JEPS meeting the journal’s Stage 1 criteria
• JEPS editor uses TESS reviews to reach decision of in-principle acceptance
• Author submits the Stage 1 manuscript to a registry (e.g., OSF)
• TESS fields the data
• Author writes up complete manuscript and resubmits to JEPS
• JEPS editor recruits initial TESS reviewers for Stage 2 review focused on compliance with the pre-registered analysis plan
• Manuscript accepted as a Registered Report