aggreCAT: An R Package for Mathematically Aggregating Expert Judgments

Abstract

Structured protocols, such as the IDEA protocol, may be used to elicit expert judgments in the form of subjective probabilities from multiple experts. Judgments from individual experts about a particular phenomena must therefore be mathematically aggregated into a single prediction. The process of aggregation may be complicated when uncertainty bounds are elicited with a judgment, and also when there are several rounds of elicitation. This paper presents the new R package \pkg{aggreCAT}, which provides 22 unique aggregation methods for combining individual judgments into a single, probabilistic measure. The aggregation methods were developed as a part of the Defense Advanced Research Projects Agency (DARPA) ‘Systematizing Confidence in Open Research and Evidence’ (SCORE) programme, which aims to generate confidence scores or estimates of ‘claim credibility’ for 3000 research claims from the social and behavioural sciences. We provide several worked examples illustrating the underlying mechanics of the aggregation methods. We also describe a general workflow for using the software in practice to facilitate uptake of this software for appropriate use-cases.

IDEAcology: An interface to streamline and facilitate efficient, rigorous expert elicitation in ecology

“…Instead, these can be calculated off‐platform using specialised software (e.g. Excalibur: Hemming, Armstrong, et al, 2020; aggreCAT: Gould et al, 2021).…”

“…Three distinct stages in the IDEAcology procedure: (a) Pre-elicitation handles the administration and planning tools in the expert elicitation; (b) the Elicitation stage handles the key steps of the IDEA protocol; (c) the Post-elicitation stage handles the Aggregation of expert estimates and prepares data for analysis. package (aggreCAT; Gould et al, 2021) and R code currently exist to do this (Hemming, 2019), but the code almost always needs to be adapted to the unique design of the elicitation requirements (e.g. to align with different data exportation formats of online survey platforms).…”

“…Instead, data are typically required to be downloaded from the platform and manually processed, which can be time consuming and prone to errors. An R package (aggreCAT; Gould et al, 2021) and R code currently exist to do this (Hemming, 2019), but the code almost always needs to be adapted to the unique design of the elicitation requirements (e.g. to align with different data exportation formats of online survey platforms).…”