Initial evidence of research quality of registered reports compared with the standard publishing model

Abstract

research question and the quality of the methodology, not whether the findings are positive, novel, and clean.More than 250 journals have adopted RRs since 2013 on the theorized promise of improving rigor and credibility. Initial evidence suggests that RRs are (1) effective at mitigating publication bias with a sharp increase in publishing negative results compared to the standard model 26,27 , and (2) cited as often or even more than other articles in the same journals 28 .However, there is no evidence about whether scholars perceive RRs to have higher, lower, or similar research quality compared with papers published in the standard model. The RR format could also have costs such as authors pursuing less interesting questions or conducting less novel or creative research 29,30 .We conducted an observational investigation of perceptions of the quality and importance of RRs compared to the standard model across a variety of outcome criteria. We recruited 353 researchers to each peer review a pair of papers, one from 29 RRs from psychology and neuroscience and one from 57 matched non-RR comparison papers.Comparison papers addressed similar topics, about half were by the same first or corresponding authors and about half were published in the same journal. RRs is a popular format for replication studies 3,31 , but replications are rare in the standard model so we excluded replication RRs. Researchers were assigned to papers according to their self-reported expertise based on the papers' keywords. Researchers self-reported that they were qualified to review the papers on average (N=353; RR M=3.74, SD=1.02; Comparison paper M=3.59, SD=1.07; Range 1 [not at all qualified] to 5 [substantially qualified]). Reviewers evaluated 19 outcome criteria including quality, rigor, novelty, creativity, and importance of the methodology and outcomes of the papers. In some RRs, authors submitted preliminary studies as initial evidence supporting the approach of the proposed last study that was peer reviewed before the findings were known. If Supplementary Table 10Article keywords included in the survey sample.

Conceptual replication study of fifteen JDM effects: Insights from the Polish sample

“…If authors follow the accepted plan, their article is published even when their results do not support the hypotheses. The study by Soderberg et al (2021) showed that Registered Reports were assessed by experts as better in all criteria (from the rigor of methodology and results to novelty and creativity) when compared with other (non-registered) papers.…”

Improving Pedagogy through Registered Reports

“…For many years the typical process has been to run a study, write-up the outcome, and hope to find a home for your work in one of the myriad of publishing outlets. However, taking our own field of Psychology as an example, as highlighted in the fallout of the Reproducibility Crisis (Soderberg et al, 2021), this standard route can lead to an increase in the quantity of publications but not necessarily an increase in quality, with approximately two-thirds of replicated studies failing to show the same effect as in the original study. This would not be such an important issue if it were not for the fact that approaches to teaching are borne out of these findings; Hansford (2021), for example, highlights a variety of measures implemented in pedagogical settings.…”

“…Therefore, even within replications, there is an issue of transparency of methods that may affect the outcome of the study and, as such, the research we base pedagogy on exhibits the same challenges of scientific credibility as other fields highlighted in the replication crisis (Soderberg et al, 2021;Munafò et al, 2017; In turn, and if the solution is not solely via replication, then the field as a whole would benefit from approaches that allow researchers to adopt norms related to transparency of methods, and to participate in practices that improve the quality and quantity of the evidence we base our teaching approaches on (Vazire, 2018).…”

Trinity Review: Integrating Registered Reports with research ethics and funding reviews

“…During Stage 1, the value of the research question, the rationale for the hypothesis, and the validity of the methods are assessed [1]. This allows reviewers to assist authors in improving the protocol and rationale, or to make changes that will further improve the quality of the paper [3]. As a result of peer review and revision, if the protocol is judged to be publishable, it is accepted in principle for publication, regardless of the results of experiments performed afterward.…”