Using meta-predictions to identify experts in the crowd when past performance is unknown

Abstract

A common approach to improving probabilistic forecasts is to identify and leverage the forecasts from experts in the crowd based on forecasters' performance on prior questions with known outcomes. However, such information is often unavailable to decision-makers on many forecasting problems, and thus it can be difficult to identify and leverage expertise. In the current paper, we propose a novel algorithm for aggregating probabilistic forecasts using forecasters' meta-predictions about what other forecasters will predict. We test the performance of an extremised version of our algorithm against current forecasting approaches in the literature and show that our algorithm significantly outperforms all other approaches on a large collection of 500 binary decision problems varying in five levels of difficulty. The success of our algorithm demonstrates the potential of using meta-predictions to leverage latent expertise in environments where forecasters' expertise cannot otherwise be easily identified.

Hidden Experts in the Crowd: Using Meta-Predictions to Leverage Expertise in Single-Question Prediction Problems

“…The weights used by the SC algorithm have useful properties relating to forecasters' expertise, but importantly, the properties of these weights are not fundamentally tied to each algorithm. Thus, the weights of the SC algorithm can be used independently, for example, for the purposes of improving forecasts in the probabilistic domain (Martinie et al 2020), or for other purposes such as identifying high-performing individuals for the purposes of compensation or evaluation.…”

“…As our outcome space is binary, it is without loss of generality that we normalise the signals so that their value is equal to the posterior belief that an event is 3 The weights used in the SC algorithm can also be used in probabilistic forecasting problems. See Martinie et al (2020) for a discussion of how the weights of the SC algorithm can be adapted to the probabilistic forecasting domain and for a comparison of the algorithm to other probabilistic forecasting algorithms proposed by Palley and Soll (2018) and Satopää et al (2016). McCoy and Prelec (2017) develops an alternative Bayesian hierarchical model that can be used in forecasting problems with multiple-choice answers.…”

Skew-Adjusted Extremized-Mean: A Simple Method for Identifying and Learning From Contrarian Minorities in Groups of Forecasters

“…The tendency towards a contrarian minority is more visible in questions 3, 5, and 7. Martinie et al (2020) rated their questions from level 1 to 5 in terms of difficulty. The easiest, level-1 questions include statements such as 'The moon shines at night because it reflects light from the sun,' whereas the hardest, level-5 questions include statements such as 'Microwaves contain more energy than visible light'.…”

Vox Peritorum: Capitalizing on Confidence and Projection to Characterize Expertise

“…Beyond political elections, there are myriad contexts for crowdsourcing judgments and decisions, which may serve as testing grounds for innovative methods that integrate available information in many different ways (e.g., Dredze et al, 2008;Hertwig, 2012;Wang et al, 2012). Of these methods, the majority rule (e.g., Hastie & Kameda, 2005), confidence weighing (e.g., Bahrami et al, 2010;Koriat, 2012ab) and meta-prediction models (e.g., Martinie et al, 2020;Palley & Satopää, 2020;Prelec et al, 2017) have received much research attention. How can these methods be used for further progress towards a deeper understanding of expert judgment?…”

“…This is where the issue of expertise provides traction. As shown in Table 1, some theories of expertise assume that the most skilled individuals project most accurately (Martinie et al, 2020;Palley and Satopää, 2020). That is, experts might show adequate levels of projection where the less skilled tend to overproject.…”