Model Uncertainty, Data Mining and Statistical Inference

Abstract

This paper takes a broad, pragmatic view of statistical inference to include all aspects of model formulation. The estimation of model parameters traditionally assumes that a model has a prespecified known form and takes no account of possible uncertainty regarding the model structure. This implicitly assumes the existence of a 'true' model, which many would regard as a fiction. In practice model uncertainty is a fact of life and likely to be more serious than other sources of uncertainty which have received far more attention from statisticians. This is true whether the model is specified on subject-matter grounds or, as is increasingly the case, when a model is formulated, fitted and checked on the same data set in an iterative, interactive way. Modern computing power allows a large number of models to be considered and data-dependent specification searches have become the norm in many areas of statistics. The term data mining may be used in this context when the analyst goes to great lengths to obtain a good fit. This paper reviews the effects of model uncertainty, such as too narrow prediction intervals, and the non-trivial biases in parameter estimates which can follow data-based modelling. Ways of assessing and overcoming the effects of model uncertainty are discussed, including the use of simulation and resampling methods, a Bayesian model averaging approach and collecting additional data wherever possible. Perhaps the main aim of the paper is to ensure that statisticians are aware of the problems and start addressing the issues even if there is no simple, general theoretical fix.

Deforestation modelling using logistic regression and GIS

“…80% of observations were allocated to establish the model, and 20% of them were allocated to the validation model. The model has been calibrated using a separate dataset, to reduce the likelihood of overestimating the predictive ability of the model (Chatfield 1995;Wilson et al 2005).…”

Forecasting Employment in Europe: Are Survey Results Helpful?

“…From Table 3 we additionally can conclude that there are differences between the forecasting performance of EEXP_av and EEXP_tm 15 One would argue that adding an indicator and therefore getting a better in-sample fit for the data has to result in a better out-of-sample performance. This may not be the case (see Chatfield, 1995). Overfitting the model or parameter instabilities (see Rossi/Sekhposyan, 2011) are some explanations why in-sample and out-of-sample performance may differ.…”

Decentralization as an Institutional System, Prospects and Retrospect: Case of Four Weredas of Tigrai

“…6 For a simple one-way MANOVA, the data set should have one independent variable (grouping variable) and at least two dependent variables. If the model is a type in which it is assumed that two factor determines the mean value of a variable it is called two-factor analysis of variance, two ways MANOVA (Chatfield, 1995). ANOVA tests for differences among group means, and assumes that the mean of a variable depends on only one factor, namely the sample from which the observation is taken.…”