Model Selection: An Integral Part of Inference

Abstract

We argue that model selection uncertainty should be fully incorporated into statistical inference whenever estimation is sensitive to model choice and that choice is made with reference to the data. We consider different philosophies for achieving this goal and suggest strategies for data analysis. We illustrate our methods through three examples. The first is a Poisson regression of bird counts in which a choice is to be made between inclusion of one or both of two covariates. The second is a line transect data set for which different models yield substantially different estimates of abundance. The third is a simulated example in which truth is known.

Epidemiology of Chronic Wasting Disease in Free-Ranging Mule Deer: Spatial, Temporal, and Demographic Influences on Observed Prevalence Patterns

“…We used Akaike's Information Criterion (Akaike, 1973) corrected for small sample bias (AICc) as the basis for objectively ranking models and selecting an appropriate ''best approximating'' model (Burnham and Anderson, 2002). We ranked and compared models using ⌬AICc (Leberton et al, 1992;Burnham and Anderson, 2002) and normalized AICc weights (ŵ m ) (Buckland et al, 1997;Burnham and Anderson, 2002). Models that were Յ2 AICc units removed from the best model were considered to be competing models.…”

A novel approach to the bias-variance problem in bump hunting

“…(step) a stepwise-in approach with c = 0, where the background model is initially chosen as only the = 0 mode, and higher-order modes are added successively but only if ∆Λ > 1; 8 (aic) the standard AIC approach, where background models with max from 0 to 10 are considered and the one that minimizes Λ with c = 2 is selected;…”

“…The variation considered here defines a weight for each of the 64 models w m ∝ exp(−Λ m /2), where m denotes the model and Λ is defined using c = 2 (this results in the exponent being half the AIC value for each model); the weights are then normalized such that they sum to unity [8]. The signal strength is the weighted average of theŜ values obtained from each of the 64 modelsŜ avg = ∑ w mŜm .…”

Quantitative Detection of Cirrhosis: Towards the Development of Computer-Assisted Detection Method

“…The next four selected models according to their AUC performance are listed thereafter on Table 2. Finally, we also averaged all top 100 selected models, where the method of weighting different models is based on Buckland et al [17]. The AUC performance for model averaging was 0.88 and 0.87 for training set and validation set respectively (Table 2) We ranked all the morphological features by the frequency of occurrence in our top 100 models (Fig.…”