Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process

Abstract

As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce the repliCATS (Collaborative Assessments for Trustworthy Science) process, a new method for eliciting expert predictions about the replicability of research. This process is a structured expert elicitation approach based on a modified Delphi technique applied to the evaluation of research claims in social and behavioural sciences. The utility of processes to predict replicability is their capacity to test scientific claims without the costs of full replication. Experimental data supports the validity of this process, with a validation study producing a classification accuracy of 84% and an Area Under the Curve of 0.94, meeting or exceeding the accuracy of other techniques used to predict replicability. The repliCATS process provides other benefits. It is highly scalable, able to be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period through an online elicitation platform, having been used to assess 3000 research claims over an 18 month period. It is available to be implemented in a range of ways and we describe one such implementation. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to provide insight in understanding the limits of generalizability of scientific claims. The primary limitation of the repliCATS process is its reliance on human-derived predictions with consequent costs in terms of participant fatigue although careful design can minimise these costs. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies.

The wisdom of the coherent: Improving correspondence with coherence-weighted aggregation.

“…Although we have emphasized the utility of coherence-weighting as a tool that can be used even in data-poor environments, nothing prevents decision makers from combining it with other techniques to further improve the accuracy of aggregated estimates such as other performance-weighted methods (Bolger & Rowe, 2015; Budescu & Chen, 2015; Clemen & Winkler, 1999; Collins et al, in press; Himmelstein et al, 2021). Furthermore, practitioners can combine ensembles of methods such as competitive (Lichtendahl et al, 2013) or structured (Fraser et al, 2023) elicitation methods; enhancing the salience of private versus public information (Larrick et al, 2012), choosing smaller, wiser crowds (Soll et al, 2010); trimming opinion pools to account for under- and overconfidence (Jose et al, 2014; Yaniv, 1997); up-weighting assessors who update estimates frequently in small increments (Atanasov et al, 2020); or extremizing judgments (Baron et al, 2014; Hanea et al, 2021; Satopää & Ungar, 2015) to further improve accuracy. The latter is particularly appealing in general-knowledge tasks where the “outcomes” are, by definition, extreme.…”

Predicting the replicability of social and behavioural science claims in COVID-19 preprints

“…We recruited 99 experts from a pool of people who had previously participated in at least one repliCATS workshop or remote process for evaluating research claims, described in Fraser et al (2023). Each expert participant was awarded a $200USD grant to assess 10 research claims for this study.…”

“…Participants evaluated their assigned claims on an online platform developed for the repliCATS project (Pearson 2020;Fraser et al 2023;Fig M1) implementing the 'IDEA' protocol (Hanea et al, 2017;Hemming et al, 2018a;Hemming et al, 2018b, Fig. 1).…”

“…Third, we directly compared the performance of an expert sample to a novice one. Finally, we directly compared predictions made using a cooperative interactive Delphi-style structured process with two elicitation rounds (Fraser et al, 2023; Figure 1) against the outcomes of a prediction market. With this study we also release a new corpus of 35 replications -The COVID-19 Preprint Replication Project.…”

Predicting and reasoning about replicability using structured groups

“…Details of these prompts are given in the electronic supplementary material. Data were analysed using a subset of analytic categories (codes) developed through qualitative content-analysis techniques during the repliCATS project [31,47]. These predefined codes were collected in the 'Known-Outcome Codebook', along with inclusion and exclusion criteria, to guide analysts in interpreting text instances with respect to relevant codes.…”