Model selection procedures in social research: Monte-Carlo simulation results

Raffalovich, Lawrence E.; Deane, Glenn; Armstrong, David J.; Tsao, Hui-shien

doi:10.1080/03081070802203959

Cited by 48 publications

(32 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This led Burnham & Anderson () to state that one should use ‘AIC for tapering effects and BIC for when there are no effects at all or a few big effects and all others are zero effects (no intermediate effects, no tapering effects)’, although this is not a universally held view: Raffalovich et al . () claimed via simulation experiments that BIC was generally preferable in all cases. We examine this issue closely in a simulation study.…”

Section: Introductionmentioning

confidence: 96%

The relative performance of AIC, AIC_Cand BIC in the presence of unobserved heterogeneity

2016

View full text Add to dashboard Cite

Summary Model selection is difficult. Even in the apparently straightforward case of choosing between standard linear regression models, there does not yet appear to be consensus in the statistical ecology literature as to the right approach. We review recent works on model selection in ecology and subsequently focus on one aspect in particular: the use of the Akaike Information Criterion (AIC) or its small‐sample equivalent, AICC. We create a novel framework for simulation studies and use this to study model selection from simulated data sets with a range of properties, which differ in terms of degree of unobserved heterogeneity. We use the results of the simulation study to suggest an approach for model selection based on ideas from information criteria but requiring simulation. We find that the relative predictive performance of model selection by different information criteria is heavily dependent on the degree of unobserved heterogeneity between data sets. When heterogeneity is small, AIC or AICC are likely to perform well, but if heterogeneity is large, the Bayesian Information Criterion (BIC) will often perform better, due to the stronger penalty afforded. Our conclusion is that the choice of information criterion (or more broadly, the strength of likelihood penalty) should ideally be based upon hypothesized (or estimated from previous data) properties of the population of data sets from which a given data set could have arisen. Relying on a single form of information criterion is unlikely to be universally successful.

show abstract

Section: Introductionmentioning

confidence: 96%

The relative performance of AIC, AIC_Cand BIC in the presence of unobserved heterogeneity

2016

View full text Add to dashboard Cite

show abstract

“…2007; Ward 2008; Lee & Ghosh 2009). Raffalovich et al. (2008) provide an excellent review of work done in this area.…”

Section: Introductionmentioning

confidence: 99%

Performance of several variable‐selection methods applied to real ecological data

Murtaugh¹

2009

Ecology Letters

315

228

View full text Add to dashboard Cite

I evaluated the predictive ability of statistical models obtained by applying seven methods of variable selection to 12 ecological and environmental data sets. Cross-validation, involving repeated splits of each data set into training and validation subsets, was used to obtain honest estimates of predictive ability that could be fairly compared among methods. There was surprisingly little difference in predictive ability among five methods based on multiple linear regression. Stepwise methods performed similarly to exhaustive algorithms for subset selection, and the choice of criterion for comparing models (Akaike's information criterion, Schwarz's Bayesian information criterion or F statistics) had little effect on predictive ability. For most of the data sets, two methods based on regression trees yielded models with substantially lower predictive ability. I argue that there is no 'best' method of variable selection and that any of the regression-based approaches discussed here is capable of yielding useful predictive models.

show abstract

“…the best model). We did not correct AIC for small sample size because analyses suggest that such correction does not improve model selection (Richards, 2005; Raffalovich et al. , 2008).…”

Section: Methodsmentioning

confidence: 99%

Ecogeographic variation of body size in the spectacled salamanders (Salamandrina): influence of genetic structure and local factors

Romano

Ficetola

2010

Journal of Biogeography

View full text Add to dashboard Cite

Aim The patterns and causes of ecogeographical body size variation in ectotherms remain controversial. In amphibians, recent genetic studies are leading to the discovery of many cryptic species. We analysed the relationships between body size and climate for a salamander (Salamandrina) that was recently separated into two sibling species, to evaluate how ignoring interspecific and intraspecific genetic structure may affect the conclusions of ecogeographical studies. We also considered the potential effects of factors acting at a local scale.Location Thirty-four populations covering the whole range of Salamandrina, which is endemic to peninsular Italy.Methods We pooled original data and data from the literature to obtain information on the snout-vent length (SVL) of 3850 Salamandrina females; we obtained high-resolution climatic data from the sampled localities. We used an information-theoretic approach to evaluate the roles of climate, genetic features (mitochondrial haplogroup identity) and characteristics of aquatic oviposition sites. We repeated our analyses three times: in the first analysis we ignored genetic data on intraspecific and interspecific variation; in the second one we considered the recently discovered differences between the two sibling species; in the third one we included information on intraspecific genetic structure within Salamandrina perspicillata (for Salamandrina terdigitata the sample size was too small to perform intraspecific analyses).Results If genetic information was ignored, our analysis suggested the existence of a relationship between SVL and climatic variables, with populations of large body size in areas with high precipitation and high thermal range. If species identity was included in the analysis, the role of climatic features was much weaker. When intraspecific genetic differences were also considered, no climatic feature had an effect. In all analyses, local factors were important and explained a large proportion of the variation; populations spawning in still water had a larger body size.Main conclusions An imperfect knowledge of species boundaries, or overlooking the intraspecific genetic variation can strongly affect the results of analyses of body size variation. Furthermore, local factors can be more important than the large-scale parameters traditionally considered, particularly in species with a small range.

show abstract

Model selection procedures in social research: Monte-Carlo simulation results

Cited by 48 publications

References 14 publications

The relative performance of AIC, AIC_Cand BIC in the presence of unobserved heterogeneity

The relative performance of AIC, AIC_Cand BIC in the presence of unobserved heterogeneity

Performance of several variable‐selection methods applied to real ecological data

Ecogeographic variation of body size in the spectacled salamanders (Salamandrina): influence of genetic structure and local factors

Contact Info

Product

Resources

About

Model selection procedures in social research: Monte-Carlo simulation results

Cited by 48 publications

References 14 publications

The relative performance of AIC, AICCand BIC in the presence of unobserved heterogeneity

The relative performance of AIC, AICCand BIC in the presence of unobserved heterogeneity

Performance of several variable‐selection methods applied to real ecological data

Ecogeographic variation of body size in the spectacled salamanders (Salamandrina): influence of genetic structure and local factors

Contact Info

Product

Resources

About

The relative performance of AIC, AIC_Cand BIC in the presence of unobserved heterogeneity

The relative performance of AIC, AIC_Cand BIC in the presence of unobserved heterogeneity