Predicting reliability through structured expert elicitation with repliCATS (Collaborative Assessments for Trustworthy Science)

Fraser, Hannah; Bush, Martin; Wintle, Bonnie; Mody, Fallon; Smith, Eden T.; Hanea, Anca M.; Gould, Elliot; Hemming, Victoria; Hamilton, Daniel G; Rumpff, Libby; Wilkinson, David P.; Pearson, Robert A.; Thorn, Felix Singleton; Ashton, raquel; Willcox, Aaron; Gray, Charles T.; Head, Andrew; Ross, Melissa; Groenewegen, Rebecca; Marcoci, Alexandru; Vercammen, Ans; Parker, Timothy; Hoekstra, Rink; Nakagawa, Shinichi; Mandel, David R.; Ravenzwaaij, Don van; McBride, Marissa; Sinnott, Richard; Vesk, Peter A.; Burgman, Mark A.; Fidler, Fiona

doi:10.31222/osf.io/2pczv

Cited by 9 publications

(7 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In repliCATS, experts work in small groups of 4 to 6 people using a custom built cloud-based elicitation platform (Fraser et al, 2021;Pearson et al, 2021). Each group is provided with a paper to read and a specific claim from the paper to assess.…”

Section: Figure 3 Overview Of the Idea Protocol As Adopted In The Replicats Projectmentioning

confidence: 99%

Systematizing Confidence in Open Research and Evidence (SCORE)

Alipourfard¹,

Arendt²,

Benjamin³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Assessing the credibility of research claims is a central, continuous, and laborious part of the scientific process. Credibility assessment strategies range from expert judgment to aggregating existing evidence to systematic replication efforts. Such assessments can require substantial time and effort. Research progress could be accelerated if there were rapid, scalable, accurate credibility indicators to guide attention and resource allocation for further assessment. The SCORE program is creating and validating algorithms to provide confidence scores for research claims at scale. To investigate the viability of scalable tools, teams are creating: a database of claims from papers in the social and behavioral sciences; expert and machine generated estimates of credibility; and, evidence of reproducibility, robustness, and replicability to validate the estimates. Beyond the primary research objective, the data and artifacts generated from this program will be openly shared and provide an unprecedented opportunity to examine research credibility and evidence.

show abstract

Section: Figure 3 Overview Of the Idea Protocol As Adopted In The Replicats Projectmentioning

confidence: 99%

Systematizing Confidence in Open Research and Evidence (SCORE)

Alipourfard¹,

Arendt²,

Benjamin³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The repliCATS project adapted and deployed the IDEA protocol to elicit crowd-sourced judgments from diverse groups about the likely replicability of SBS research claims (Fraser et al 2021). The IDEA ('Investigate,' 'Discuss,' 'Estimate' and 'Aggregate') protocol is a structured elicitation protocol that draws on the 'wisdom of crowds' to elicit subjective judgments about the likelihood of uncertain events.…”

Section: Darpa Score Program and The Replicats Projectmentioning

confidence: 99%

aggreCAT: An R Package for Mathematically Aggregating Expert Judgments

Gould¹,

Gray²,

Groenewegen³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Structured protocols, such as the IDEA protocol, may be used to elicit expert judgments in the form of subjective probabilities from multiple experts. Judgments from individual experts about a particular phenomena must therefore be mathematically aggregated into a single prediction. The process of aggregation may be complicated when uncertainty bounds are elicited with a judgment, and also when there are several rounds of elicitation. This paper presents the new R package \pkg{aggreCAT}, which provides 22 unique aggregation methods for combining individual judgments into a single, probabilistic measure. The aggregation methods were developed as a part of the Defense Advanced Research Projects Agency (DARPA) ‘Systematizing Confidence in Open Research and Evidence’ (SCORE) programme, which aims to generate confidence scores or estimates of ‘claim credibility’ for 3000 research claims from the social and behavioural sciences. We provide several worked examples illustrating the underlying mechanics of the aggregation methods. We also describe a general workflow for using the software in practice to facilitate uptake of this software for appropriate use-cases.

show abstract

“…Formal structured elicitation protocols, such as the one used here, have been developed to minimise the limitations and associated biases of expert judgments (Cooke 1991;Morgan & Henrion 1990;O'Hagan et al 2006). Even though a longer elicitation process is associated with declining quantity and quality of information, it appears that the benefits of following a structured protocol outweigh the potential drawbacks (Fraser et al 2021). Finally, even though our expert-elicited interaction strengths would have benefited from an increased number of responses for certain interactionsas a larger number of responses is generally associated with less bias when aggregating (O'Hagan et al 2006)we can still be confident in our assessment.…”

Section: Limitations Of Expert Knowledge Elicitationmentioning

confidence: 99%

Quantifying Earth system interactions for sustainable food production: an expert elicitation

Chrysafi

Virkki

Jalava

et al. 2021

Preprint

View full text Add to dashboard Cite

Several safe boundaries of critical Earth system processes have already been crossed by human perturbations. Recent research indicates that not accounting for the interactions between these processes may further narrow the safe operating space for humanity. Yet existing work accounts only for transgression of single boundaries and only a few studies take some of the boundary interactions into account. For future sustainability assessments, it is essential to understand boundary transgressions and their interactions more comprehensively. Here, we explore quantitatively how strongly seven variables, representing Earth system processes relevant to food production, interact with each other, using a structured expert knowledge elicitation. We identify Green water and Land system change as crucial interactive processes through their impacts on multiple relevant processes, while Biosphere integrity-land, freshwater and ocean components appear to be most affected by other Earth system processes, most notably Blue water and Biogeochemical flows. The elicitation also enabled us to map the complex network of mechanisms mediating interactions, to support integrated Earth system and planetary boundaries modelling and assessments. Finally, we created a prioritisation scheme for future research according to the interaction strengths and existing knowledge gaps. Our analysis improves our understanding of Earth system interactions, with clear implications for sustainable use of natural resources such as the biophysical limits for food production.

show abstract

Predicting reliability through structured expert elicitation with repliCATS (Collaborative Assessments for Trustworthy Science)

Cited by 9 publications

References 38 publications

Systematizing Confidence in Open Research and Evidence (SCORE)

Systematizing Confidence in Open Research and Evidence (SCORE)

aggreCAT: An R Package for Mathematically Aggregating Expert Judgments

Quantifying Earth system interactions for sustainable food production: an expert elicitation

Contact Info

Product

Resources

About