Coherent combination of experts' opinions

Dawid, A. Philip; DeGroot, Morris H.; Mortera, Julia; Cooke, Roger M.; French, Simon; Genest, Christian; Schervish, Mark J.; Lindley, D. V.; McConway, Kevin; Winkler, Robert L.

doi:10.1007/bf02562628

Cited by 72 publications

(79 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…arithmetic mean, but there are many reasonable alternatives, giving ample room for adjustments or "tuning" (27)(28)(29)(30).] In our case, the arithmetic mean performs poorly, as we have validated by comparing its distance to the truth with the individual distances to the truth.…”

Section: Resultsmentioning

confidence: 78%

How social influence can undermine the wisdom of crowd effect

Lorenz

Rauhut

Schweitzer

et al. 2011

Proc. Natl. Acad. Sci. U.S.A.

844

828

View full text Add to dashboard Cite

Social groups can be remarkably smart and knowledgeable when their averaged judgements are compared with the judgements of individuals. Already Galton [Galton F (1907) Nature 75:7] found evidence that the median estimate of a group can be more accurate than estimates of experts. This wisdom of crowd effect was recently supported by examples from stock markets, political elections, and quiz shows [Surowiecki J (2004) The Wisdom of Crowds]. In contrast, we demonstrate by experimental evidence (N = 144) that even mild social influence can undermine the wisdom of crowd effect in simple estimation tasks. In the experiment, subjects could reconsider their response to factual questions after having received average or full information of the responses of other subjects. We compare subjects' convergence of estimates and improvements in accuracy over five consecutive estimation periods with a control condition, in which no information about others' responses was provided. Although groups are initially "wise," knowledge about estimates of others narrows the diversity of opinions to such an extent that it undermines the wisdom of crowd effect in three different ways. The "social influence effect" diminishes the diversity of the crowd without improvements of its collective error. The "range reduction effect" moves the position of the truth to peripheral regions of the range of estimates so that the crowd becomes less reliable in providing expertise for external observers. The "confidence effect" boosts individuals' confidence after convergence of their estimates despite lack of improved accuracy. Examples of the revealed mechanism range from misled elites to the recent global financial crisis.collective judgment | estimate aggregation | experimental social science | swarm intelligence | overconfidence

show abstract

Section: Resultsmentioning

confidence: 78%

How social influence can undermine the wisdom of crowd effect

Lorenz

Rauhut

Schweitzer

et al. 2011

Proc. Natl. Acad. Sci. U.S.A.

844

828

View full text Add to dashboard Cite

show abstract

“…First, nothing has been said in this paper about other forms of averaging or indeed forms of opinion pooling that do not involve averaging. Considerable guidance on this question can be found in Dawid et al (1995), where a much more extensive set of formal results on the Bayes compatibility of opinion pooling is proved. The upshot of these results is far from settled, however, and the philosophical status of other pooling rules deserve further exploration.…”

Section: Discussionmentioning

confidence: 99%

“…In subsequent sections, I investigate what Bayesianism requires of agents in some rather simple situations; in particular, ones which mandate deference to the opinions of one or more experts on some specific proposition. Finally I consider the consistency of these requirements with those implied by linear averaging, drawing particularly on Dawid et al (1995). The conclusion is somewhat surprising.…”

Section: Introductionmentioning

confidence: 94%

Learning from others: conditioning versus averaging

Bradley

2017

Theory Decis

View full text Add to dashboard Cite

How should we revise our beliefs in response to the expressed probabilistic opinions of experts on some proposition when these experts are in disagreement? In this paper I examine the suggestion that in such circumstances we should adopt a linear average of the experts' opinions and consider whether such a belief revision policy is compatible with Bayesian conditionalisation. By looking at situations in which full or partial deference to the expressed opinions of others is required by Bayesianism I show that only in trivial circumstances are the requirements imposed by linear averaging compatible with it.

show abstract

“…We would expect interesting relations whose individual components (e.g., entities and events) were identified with a higher score, to stand a higher chance of being true. To this end, one can employ confidence mixtures (Iversen et al, 2008;Dawid et al, 1995). Given k experts, each returning a confidence value c i ∈ R, i = 1 .…”

Section: Evidence Sources For Confidencementioning

confidence: 99%

“…Confidence aggregation Different confidence sources will have a different impact on their global aggregate (Iversen et al, 2008;Dawid et al, 1995). Such impact can be quantified as a weight, set a priori or a posteriori by training a classifier over gold (or silver) data and plugging into the aggregates the inferred weights (Liu et al, 2012).…”

Section: Component-wise Confidencementioning

confidence: 99%

Towards Confidence Estimation for Typed Protein-Protein Relation Extraction

Thorne¹,

Klinger²

2017

Proceedings of the Biomedical NLP Workshop

View full text Add to dashboard Cite

Systems which build on top of information extraction are typically challenged to extract knowledge that, while correct, is not yet well-known. We hypothesize that a good confidence measure for relational information has the property that such interesting information is found between information extracted with very high confidence and very low confidence. We discuss confidence estimation for the domain of biomedical protein-protein relation discovery in biomedical literature. As facts reported in papers take some time to be validated and recorded in biomedical databases, such task gives rise to large quantities of unknown but potentially true candidate relations. It is thus important to rank them based on supporting evidence rather than discard them. In this paper, we discuss this task and propose different approaches for confidence estimation and a pipeline to evaluate such methods. We show that the most straight-forward approach, a combination of different confidence measures from pipeline modules seems not to work well. We discuss this negative result and pinpoint potential future research directions.

show abstract

Coherent combination of experts' opinions

Abstract: Expert Opinions, Coherence, Compatibility, Combining Opinions, Linear Opinion Pool, Logarithmic Opinion Pool, Harmonic Opinion Pool,

Cited by 72 publications

References 30 publications

How social influence can undermine the wisdom of crowd effect

How social influence can undermine the wisdom of crowd effect

Learning from others: conditioning versus averaging

Towards Confidence Estimation for Typed Protein-Protein Relation Extraction

Contact Info

Product

Resources

About