Aggregating predictions from experts: A review of statistical methods, experiments, and applications

McAndrew, Thomas; Wattanachit, Nutcha; Gibson, Graham Casey; Reich, Nicholas G.

doi:10.1002/wics.1514

Cited by 36 publications

(32 citation statements)

References 154 publications

(463 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In this context, our results demonstrate that it is preferable to use an equally-weighted ensemble to buffer against uncertainty in optimal ensemble weights. As is also being demonstrated in forecasts of COVID-19, equally weighted ensembles can provide accurate forecasts [26, 45] and may be a better reflection of the considerable structural uncertainty inherent to models of emerging pathogen transmission [24].…”

Section: Discussionmentioning

confidence: 99%

Trade-offs between individual and ensemble forecasts of an emerging infectious disease

Oidtman¹,

Omodei²,

Kraemer³

et al. 2021

Preprint

View full text Add to dashboard Cite

When new pathogens emerge, numerous questions arise about their future spread, some of which can be addressed with probabilistic forecasts. The many uncertainties about the epidemiology of emerging pathogens can make it difficult to choose among model structures and assumptions, however. To assess the potential for uncertainties about emerging pathogens to affect forecasts of their spread, we evaluated the performance of a suite of 16 forecasting models in the context of the 2015-2016 Zika epidemic in Colombia. Each model featured a different combination of assumptions about the role of human mobility in driving transmission, spatiotemporal variation in transmission potential, and the number of times the virus was introduced. All models used the same core transmission model and the same iterative data assimilation algorithm to generate forecasts. By assessing forecast performance through time using logarithmic scoring with ensemble weighting, we found that which model assumptions had the most ensemble weight changed through time. In particular, spatially coupled models had higher ensemble weights in the early and late phases of the epidemic, whereas non-spatial models had higher ensemble weights at the peak of the epidemic. We compared forecast performance of the equally-weighted ensemble model to each individual model and identified a trade-off whereby certain individual models outperformed the ensemble model early in the epidemic but the ensemble model outperformed all individual models on average. On balance, our results suggest that suites of models that span uncertainty across alternative assumptions are necessary to obtain robust forecasts in the context of emerging infectious diseases.

show abstract

Section: Discussionmentioning

confidence: 99%

Trade-offs between individual and ensemble forecasts of an emerging infectious disease

Oidtman¹,

Omodei²,

Kraemer³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…For example we use the posterior estimates prior to time T and forecast assuming time variable parameters were constant in the forecasting horizon and equal to me average estimated for the previous week t ∈ [ T − 10, T ] this forecast reasoning have also seen in [26] for more details in the forecast methods see section S.5 in Supplementary Material. Then we evaluated the forecast performance using different scores to measure the fit to the observations [27, 28, 29]. We implemented a probabilistic assessment of our forecasts calculating the sharpness of predictive distributions subject to calibration [29, 30].…”

Section: Methodsmentioning

confidence: 99%

Estimating and forecasting the burden and spread of SARS-CoV2 first wave in Colombia

Vega

Enrique

2021

Preprint

View full text Add to dashboard Cite

Following the rapid dissemination of COVID-19 cases in Colombia in 2020, large-scale non-pharmaceutical interventions (NPIs) were implemented as national emergencies in most of the municipalities of the country starting by a lockdown on March 20th of 2020. Using combinations of meta-population models SEAIIRD (Susceptible-Exposed-Asymptomatic-Infected-Recovered-Diseased) which describes the disease dynamics in the different localities, with movement data that accounts for the number of commuters between units and statistical inference algorithms could be an effective approach to both nowcast and forecast the number of cases and deaths in the country. Here we used an iterated filtering (IF) framework to fit the parameters of our model to the reported data across municipalities from march to late October in locations with more than 50 reported deaths and cases historically. Since the model is high dimensional (6 state variable by municipality) inference on those parameters is highly non-trivial, so we used an Ensemble-Adjustment-Kalman-Filter (EAKF) to estimate time variable system states and parameters. Our results show that the model is capable of capturing the evolution of the outbreak in the country and providing estimates of the epidemiological parameters in time. These estimates could become the base for planning future interventions as well as evaluate the impact of NPIs on the effective reproductive number (ℛeff) and the key epidemiological parameters, such as the contact rate or the reporting rate. Our approach demonstrates that real-time, publicly available ensemble forecasts can provide robust short-term predictions of reported COVID-19 deaths in Colombia. This model has the potential to be used as a forecasting and prediction tool to evaluate disease dynamics and to develop a real time surveillance system for management and control.

show abstract

“…These help to improve accuracy, and reduce overconfidence by reducing biases such as anchoring, groupthink, and confirmation bias. The IDEA protocol aims to improve accuracy through controlling groupthink by aggregating group assessments mathematically and not behaviourally [12][13][14][15]. That is, group members are not forced to agree on a single final judgement that reflects the whole group.…”

Section: Research Goalsmentioning

confidence: 99%

Predicting reliability through structured expert elicitation with repliCATS (Collaborative Assessments for Trustworthy Science)

Fraser¹,

Bush²,

Wintle³

et al. 2021

Preprint

View full text Add to dashboard Cite

Replication is a hallmark of scientific research. As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce a new technique to evaluating replicability, the repliCATS (Collaborative Assessments for Trustworthy Science) process, a structured expert elicitation approach based on the IDEA protocol. The repliCATS process is delivered through an underpinning online platform and applied to the evaluation of research claims in social and behavioural sciences. This process can be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period. Pilot data suggests that the accuracy of the repliCATS process meets or exceeds that of other techniques used to predict replicability. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to assist with problems like understanding the limits of generalizability of scientific claims. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies.

show abstract

Aggregating predictions from experts: A review of statistical methods, experiments, and applications

Cited by 36 publications

References 154 publications

Trade-offs between individual and ensemble forecasts of an emerging infectious disease

Trade-offs between individual and ensemble forecasts of an emerging infectious disease

Estimating and forecasting the burden and spread of SARS-CoV2 first wave in Colombia

Predicting reliability through structured expert elicitation with repliCATS (Collaborative Assessments for Trustworthy Science)

Contact Info

Product

Resources

About