Choice of generalized linear mixed models using predictive crossvalidation

Abstract

The choice of generalized linear mixed models is difficult, because it involves the selection of both fixed and random effects. Classical criteria like Akaike's information criterion (AIC) are often not suitable for the latter task, and others which are useful in linear mixed models are difficult to extend to the generalized case, especially for overdispersed data. A predictive leave-one-out crossvalidation approach is suggested that can be applied for choosing both fixed and random effects, even in models with overdispersion, and is based on proper scoring rules. An attractive feature of this approach is the fact that the model has to be fitted just once to the data set, which makes computations fast and convenient. As the calculation of the leave-one-out predictive distribution is not possible analytically, it is shown how an iteratively weighted least squares algorithm combined with some analytic approximations can be used for this task. A simulation study and two applications of the methodology to binary and count data are provided, as well as comparisons with two other methods. AbstractThe choice of generalized linear mixed models is difficult, because it involves the selection of both fixed and random effects. Classical criteria like Akaike's information criterion (AIC) are often not suitable for the latter task, and others which are useful in linear mixed models are difficult to extend to the generalized case, especially for overdispersed data. A predictive leave-one-out crossvalidation approach is suggested that can be applied for choosing both fixed and random effects, even in models with overdispersion, and is based on proper scoring rules. An attractive feature of this approach is the fact that the model has to be fitted just once to the data set, which makes computations fast and convenient. As the calculation of the leave-one-out predictive distribution is not possible analytically, it is shown how an iteratively weighted least squares algorithm combined with some analytic approximations can be used for this task. A simulation study and two applications of the methodology to binary and count data are provided, as well as comparisons with two other methods.

Marginal or conditional regression models for correlated non‐normal data?

“…On the other hand, marginalized multilevel models rely on likelihood methods and can thus undergo model selection procedures just like GLMMs. Although model choice can still be relatively challenging, valid information criteria for likelihood approaches exist, such as a modified Schwarz criterion (Pauler ), the conditional AIC (Hodges & Sargent ), a predictive cross‐validation approach (Braun, Sabanés Bové & Held ), or a generalized R 2 (Nakagawa & Schielzeth ; Johnson ).…”

Spatio‐phylogenetic multispecies distribution models

“…If several variance estimates were positive, we thus needed to choose among these by model selection. Classical criteria for model selection, such as the AIC or BIC, are not adequate for model selection in the mixed models setting when the focus is on the choice of random effects (Vaida & Blanchard 2005;Braun et al 2014). Vaida & Blanchard (2005) proposed a conditional AIC criterion for comparing linear mixed models with different random effects structures, based on inference on the conditional likelihood.…”

“…The concept was extended to also apply to GLMMs by both Yu & Yau (2012) and Saefken et al (2014). Another approach, suggested by Braun et al (2014), uses mean crossvalidated proper scores (see Gneiting & Raftery 2007) to choose the model with the best predictive abilities in the GLMM setting. This method, which we utilized for the PSGLMMs, is suitable for choosing random as well as fixed effects.…”

“…For both scores, a larger score means better predictive abilities. We refer to Braun et al (2014) for technical details. The PSGLMMs may also be used to choose among several phylogenetic trees to decide which is best suited for the observed species data.…”

“…For Gaussian responses, all discarded runs were due to the parameter estimation (in lme4) not converging due to non-identifiable variance and residual variance parameters, see Section Computational considerations. For Poisson responses for the ESM, most discarded runs were due to numerical issues pertaining to the predictive cross-validation, see Braun et al (2014).…”