Evaluating Registered Reports Funding Partnerships: a feasibility study

Abstract

Background: We studied a novel initiative – Registered Reports Funding Partnerships (RRFPs) – whereby research funders and journals partner in order to integrate their procedures for funding applications and Registered Reports submissions into one process. We investigated the feasibility of conducting a randomised controlled trial (RCT) of the impact of RRFPs on (1) research quality and (2) the efficiency of the research process, from funding to publication. Methods: We conducted 32 semi-structured interviews and follow-up questionnaires with stakeholders (funders, editors, authors, and reviewers) across six different RRFPs. Results: A RCT of RRFPs appears to be feasible in principle. The partnership concept seems worthwhile to pursue further and is adaptable to the needs of various funders and publishers, and across disciplines. Three primary outcomes of interest should be measurable, and participant randomisation could conceivably be done in a number of ways. In practice, however, the current volume of submissions going through existing partnerships is too low to support a full trial. Conclusions: Although a RCT of RRFPs is conceptually feasible, it will only be possible if organisations are willing to form new partnerships, scale up existing ones, and incorporate a trial (i.e., randomisation) into these partnerships.

Development of the Brief Open Research Survey (BORS) to measure awareness and uptake of Open Research practices

“…The need for such support from wider social structures to facilitate Open Research has been previously discussed in meta research on preregistration (Osborne & Norris, 2022) and Registered Reports , and is supported by Early Career Researchers (Kowalczyk et al, 2022;Zečević et al, 2021) and Open Research networks such as the UKRN (Stewart et al, 2022). Whilst there has been promising progress to incentivise, recognise and reward Open Research practices, including within journals (e.g., via badges; Kidwell et al, 2016) and across them (e.g., via TOP factor assessment; Nosek et al, 2016), by funders (e.g., Registered Reports funding partnerships; (Clark et al, 2021;Drax et al, 2021), and some institutions (e.g., European Universities Research practices) were seen by academics as support that would most help them to use more Open Research practices. This questionnaire can be used to collect longitudinal data to examine the trajectory of Open Research, and inform strategy to develop Open Research practices.…”

“…Educational initiatives include embedding Open Research training into undergraduate and postgraduate teaching (Azevedo et al, 2022;Egan et al, 2020; Student Initiative for Open Science, 2022) and providing banks of educational resources to facilitate this (Pownall, Azevedo, et al, 2021). Incentive initiatives include journal badges issued to papers with Preregistration, Open Data and Open Materials (Kidwell et al, 2016;Rowhani-Farid & Barnett, 2018), awards for Open Research practice (Merrett et al, 2021), and funder partnerships for Registered Reports Clark et al, 2021). Staff hiring (Khan et al, 2022) and promotion criteria (Moher et al, 2018) that explicitly value Open Research practices, quality and 'slow science' (Frith, 2020) are also being increasingly adopted.…”

A qualitative analysis of stakeholder experiences with Registered Reports Funding Partnerships

“…Our analysis also provides insights into the feasibility of the continuation and expansion of partnerships. Our companion paper on the feasibility of a RCT of partnerships discusses this in greater detail 72 .…”

“…This paper deals with the first research question: “What are the experiences of the stakeholders (authors, reviewers, journal editors, funders) in the various partnership models?”. Our companion paper, led by RC, handled our second research question that aimed to investigate various factors relating to the feasibility of a partnership RCT 72 . Note that this was a deviation from our protocol, in which we aimed to investigate the question: “What outcome measures of an RCT will be valid, reliable, feasible, acceptable and yield high completion rates?”.…”

Systemic problems require systemic solutions: the need for coordination and cooperation to improve research quality

“…There is also scope for innovation and coordination with other stakeholders. For example, Registered Reports Funding Partnerships allow for the integration of the funder review process with the journal peer review process, and encourages the uptake of the Registered Reports publishing model (where publication is decided on the basis of the importance of the research question and the robustness of the methodology, rather than on the noteworthiness of otherwise of the findings) [ 3 , 4 ].…”