Disentangling Wine Judges’ Consensus, Idiosyncratic, and Random Expressions of Quality or Preference

Abstract

Judges confer various awards on wines entered in dozens of wine competitions each year. This article employs data on blind replicates to show that those awards are based on one instance of stochastic ratings assigned by wine judges; awards based on the expected values of those stochastic ratings would be different. This article recognizes the stochastic nature of ratings and builds on the work of many others to propose and test a conditional-probability model that yields maximum-likelihood estimates of judges’ latent consensus, idiosyncratic, and random assignments of scores to wines. The exact p-value for a likelihood test of the null hypothesis that the model's results are random is less than 0.001. Applying the notion of conditional probability may lead to better methods of assigning awards to entries in wine competitions and of assessing the capabilities of wine judges. (JEL Classifications: A10, C10, C00, C12, D12)

Should Ties Be Broken in Commercial Wine Competitions? When Yes, What Method Is Practical and Defensible?

“…A probability mass function (PMF) for the distribution of ratings that judges assign to a wine appears in Table 2, Equation (1). See Mallows (1957), Marden (1995), and Alvo and Yu (2014) for applications of this functional form and Bodington (2017b) for a previous application of this form in wine scores. Equation (1) is a discrete and bounded function that reflects the stochastic nature of the judges’ scores, and a maximum likelihood estimate (MLE) of the central tendency is calculated by maximizing the log likelihood in Equation (2).…”

Multifactor and multidimensional data quality analysis of judge scoring in diving competition

“…In sports competitions, judge scoring data serve as an objective measure of an athlete’s performance level. However, research has indicated the unreliability of objective measurements ( Bodington, 2017 ; Martire and Montgomery-Farrer, 2020 ; Berg et al, 2022 ). Controversy often arises regarding the quality of judge scoring data, undermining fairness and justice in sports competitions ( Looney, 2004 ).…”

The relation between Australian wine show results and prices

“… A potential limitation of the proposed approach is it assumes the opinions of show judges or experts are random errors. Cao and Stokes () and Bodington () and others suggest that there may exist some systematic or idiosyncratic expert bias, in addition to random errors in wine assessment. We comment below on how, in our case, adding additional variables to Equation to capture systematic biases do not demonstrably affect estimates. …”