Using the Data Agreement Criterion to Rank Experts’ Beliefs

Abstract

Experts' beliefs embody a present state of knowledge. It is desirable to take this knowledge into account when doing analyses or making decisions. Yet ranking experts based on the merit of their beliefs is a difficult task. In this paper we show how experts can be ranked based on their knowledge and their level of (un)certainty. By letting experts specify their knowledge in the form of a probability distribution we can assess how accurately they can predict new data, and how appropriate their level of (un)certainty is. The expert's specified probability distribution can be seen as a prior in a Bayesian statistical setting. By extending an existing prior-data conflict measure to evaluate multiple priors, i.e. experts' beliefs, we can compare experts with each other and the data to evaluate their appropriateness. Using this method new research questions can be asked and answered, for instance: Which expert predicts the new data best? Is there agreement between my experts and the data? Which experts' representation is more valid or useful? Can we reach convergence between expert judgement and data? We provided an empirical example ranking (regional) directors of a large financial institution based on their predictions of turnover.

How the Choice of Distance Measure Influences the Detection of Prior-Data Conflict

“…In [ 7 , 9 ], the KL-divergence is used for , denoted here by . The KL-divergence expresses the loss of information that occurs when we rely on the expert prior , instead of on the posterior : …”

“…The lower part of this figure shows the prior (dashed line) and the (solid line), and the upper part of this figure shows the corresponding KL-divergence, which is equal to the highlighted area under the curve. Figure 1 b illustrates the KL-divergence between the expert prior and the posterior (see [ 9 ]). To compute the the KL-divergence in the upper part of Figure 1 b is divided by the KL-divergence in the upper part of Figure 1 a.…”

“… Illustration of the criterion of Nott et al [ 9 ], with ( a ) the KL-divergence between the posterior and expert prior and ( b ) the distribution of KL-divergences between the posterior and expert prior , for 10 5 draws Y from the prior predictive. …”

“…In the current paper, we focus on two checks, developed for the detection of prior-data conflict. First, Bousquet (see [ 8 ]; see also [ 9 ]) suggested that the expert prior be compared with a non-informative, reference prior in his “Data Agreement Criterion” (DAC). When the expert prior has a larger distance to a reference posterior—a posterior based on the data and the non-informative, reference prior—than the non-informative prior, it can be concluded that the expert and data are in conflict.…”

Correction: Veen, D.; Stoel, D.; Schalken, N.; Mulder, K.; Van de Schoot, R. Using the Data Agreement Criterion to Rank Experts’ Beliefs. Entropy 2018, 20, 592

“…In Section 2.3 [ 1 ] only there is a consistent mistake in the brackets involved in the KL functions. There is one “]” bracket too many.…”