Forecasting the Accuracy of Forecasters from Properties of Forecasting Rationales

Karvetski, Christopher W.; Meinel, Carolyn; Maxwell, Daniel T.; Lu, Yunzi; Mellers, Barb

doi:10.2139/ssrn.3779404

Cited by 7 publications

(2 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Here researchers have already made some progress. Shorter‐range studies have found better forecasters offer more dialectically complex rationales (Karvetski et al, 2021; Tetlock, 2005), are better Bayesian updaters (Mellers et al, 2017), and distinguish more degrees of uncertainty (Friedman et al, 2017).…”

Section: Discussionmentioning

confidence: 99%

Long‐range subjective‐probability forecasts of slow‐motion variables in world politics: Exploring limits on expert judgment

TETLOCK

Karvetski²,

Satopää

et al. 2023

Futures & Foresight Science

Self Cite

View full text Add to dashboard Cite

Skeptics see long‐range geopolitical forecasting as quixotic. A more nuanced view is that although predictability tends to decline over time, its rate of descent is variable. The current study gives geopolitical forecasters a sporting chance by focusing on slow‐motion variables with low base rates of change. Analyses of 5, 10, and 25‐year cumulative‐risk judgments made in 1988 and 1997 revealed: (a) specialists beat generalists at predicting nuclear proliferation but not shifting nation‐state boundaries; (b) some counterfactual interventions—for example, Iran gets the bomb before 2022—boosted experts’ edge but others—for example, nuclear war before 2022—eliminated it; (c) accuracy fell faster on topics where expertise conferred no edge in shorter‐range forecasts. To accelerate scientific progress, we propose adversarial collaborations in which clashing schools of thought negotiate Bayesian reputational bets on divisive issues and use Lakatosian scorecards to incentivize the honoring of bets.

show abstract

Section: Discussionmentioning

confidence: 99%

Long‐range subjective‐probability forecasts of slow‐motion variables in world politics: Exploring limits on expert judgment

TETLOCK

Karvetski²,

Satopää

et al. 2023

Futures & Foresight Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…More sophisticated approaches from network science could be employed to identify which diagnoses are closely related and capitalize on these insights (e.g., bipartite graphs: 44). As next steps, it could also be investigated if collective performance can be further boosted by weighing users' diagnoses according to their accuracy (45), expertise (46), similarity (47), or cognitive style (48). Furthermore, future work should incorporate insights and method from information retrieval research and cognitive science on how to aggregate and evaluate lists of retrieval results (49)(50)(51).…”

Section: R a F Tmentioning

confidence: 99%

Automating Hybrid Collective Intelligence in Open-Ended Medical Diagnostics

Kurvers¹,

Nuzzolese²,

Russo³

et al. 2023

Preprint

View full text Add to dashboard Cite

Collective intelligence has emerged as a powerful mechanism to boost decision accuracy across many domains, such as geopolitical forecasting, investment, and medical diagnostics. However, collective intelligence has been mostly applied to relatively simple decision tasks (e.g., binary classifications). Applications in more open-ended tasks with a much larger problem space, such as emergency management or general medical diagnostics, are largely lacking, due to the challenge of integrating unstandardized inputs from different crowd members. Here we present a fully-automated approach for harnessing collective intelligence in the domain of general medical diagnostics. Our approach leverages semantic knowledge graphs, natural language processing, and the SNOMED CT medical ontology to overcome a major hurdle to collective intelligence in open-ended medical diagnostics, namely to identify the intended diagnosis from unstructured text. We tested our method on 1,333 medical cases diagnosed on a medical crowdsourcing platform: The Human Diagnosis Project. Each case was independently rated by ten diagnosticians. Comparing the diagnostic accuracy of single diagnosticians with the collective diagnosis of differently-sized groups, we find that our method substantially increases diagnostic accuracy: while single diagnosticians achieved 46\% accuracy, pooling the decisions of ten diagnosticians increased this to 76\%. Improvements occurred across medical specialties, chief complaints, and diagnosticians' tenure levels. Our results show the life-saving potential of tapping into the collective intelligence of the global medical community to reduce diagnostic errors and increase patient safety.

show abstract

Decomposing the effects of crowd-wisdom aggregators: The bias–information–noise (BIN) model

Satopää

Salikhov

Mellers

2023

International Journal of Forecasting

View full text Add to dashboard Cite

Forecasting the Accuracy of Forecasters from Properties of Forecasting Rationales

Cited by 7 publications

References 45 publications

Long‐range subjective‐probability forecasts of slow‐motion variables in world politics: Exploring limits on expert judgment

Long‐range subjective‐probability forecasts of slow‐motion variables in world politics: Exploring limits on expert judgment

Automating Hybrid Collective Intelligence in Open-Ended Medical Diagnostics

Decomposing the effects of crowd-wisdom aggregators: The bias–information–noise (BIN) model

Contact Info

Product

Resources

About