Probabilistic forecasting with discrete choice models: Evaluating predictions with pseudo-coefficients of determination

Abstract

Probabilistic forecasts from discrete choice models, which are widely used in marketing science and competitive event forecasting, are often best evaluated out-of-sample using pseudo-coefficients of determination, or pseudo-R 2 s. However, there is a danger of misjudging the accuracy of forecast probabilities of event outcomes, based on observed frequencies, because of issues related to pseudo-R 2 s. First, we show that McFadden's pseudo-R 2 varies predictably with the number of alternatives in the choice set. Then we evaluate the relative merits of two methods (bootstrap and asymptotic) for estimating the variance of pseudo-R 2 s so that their values can be appropriately compared across nonnested models. Finally, in the context of competitive event forecasting, where the accuracy of forecasts has direct economic consequence, we derive new R 2 measures that can be used to assess the economic value of forecasts. Throughout, we illustrate using data drawn from UK and Ireland horse race betting markets.

Meeting Expectations at the 2016 Rio Olympic Games: Country Potential and Competitiveness

“…Literature on sports betting has shown that odds can be used to predict the outcome of horse races (Ma et al, ; Sung, McDonald, and Johnson, ), hockey (Cantinotti, Ladouceur, and Jacques, ), tennis (McHale and Morton, ; Lisi and Zanella, ), the Ryder Cup (Maher, ), the Olympics (Chen, Li, and Zeng, ), and, of course, football (Andersson, Memmert, and Popowicz, ; Boulier and Stekler, ; Forrest, Goddard, and Simmons, ; Hvattum and Arntzen, ; Khazaal et al., ; Leitner, Zeileis, and Hornik, ; Štrumbelj and Šikonja, ). Constantly changing odds in all these sports are provided by bookmakers based on a combination of their expert knowledge and the wagers of fans, who place bets in the hope of beating the odds.…”