NSF Science of Science Proposal: Randomized Trial of Registered Reports

In theory, Registered Reports eliminate publication bias against negative results because publication decisions are made without knowledge of the results; increase clarity between planned (hypothesis testing; confirmatory) and unplanned (hypothesis generating; exploratory) analyses, thereby increasing the diagnosticity of statistical inferences; and, leverage peer review expertise more effectively to foster better research methodology and more informative outcomes--regardless of whether they are positive or negative results, or are consistent or inconsistent with hypothesized outcomes. With the RRs published to date, there is observational evidence for some of these expectations. But, given the potentially transformative impact of RRs on the research process, we believe that it is essential to conduct a randomized trial to evaluate the model’s qualities. By collaborating with journals, we can conduct a randomized trial of RRs in an ecologically valid context. Authors who receive revise-and-resubmit decision letters asking for an additional study will be invited to participate in the trial. Those that opt-in will be randomized to either a status quo (i.e., standard of care) or RR condition for their resubmission with the additional study. We will examine the impact of RRs on publication (e.g., acceptance rate), research outcomes (e.g., rate of positive results), and qualities of methodology (e.g., sample size, transparency, rigor).