Inequalities between expected marginal log‐likelihoods, with implications for likelihood‐based model complexity and comparison measures

Abstract

A multi-level model allows the possibility of marginalization across levels in different ways, yielding more than one possible marginal likelihood. Since log-likelihoods are often used in classical model comparison, the question to ask is which likelihood should be chosen for a given model. The authors employ a Bayesian framework to shed some light on qualitative comparison of the likelihoods associated with a given model. They connect these results to related issues of the effective number of parameters, penalty function, and consistent definition of a likelihood-based model choice criterion. In particular, with a two-stage model they show that, very generally, regardless of hyperprior specification or how much data is collected or what the realized values are, a priori, the first-stage likelihood is expected to be smaller than the marginal likelihood. A posteriori, these expectations are reversed and the disparities worsen with increasing sample size and with increasing number of model levels

Bayesian mixed‐effects location and scale models for multivariate longitudinal outcomes: an application to ecological momentary assessment data

“…Spiegelhalter et al [54], Ntzoufras [55, Chapter 11], and Lesaffre and Lawson [56, Chapter 10] have extensively discussed differences between conditional and marginal inference for hierarchical models to explain the importance of DIC based on the conditional likelihood for the model comparison. However, Trevisani and Gelfand [57] have shown that, a posteriori, the first-stage likelihood is expected to be bigger than the marginal likelihood for any given model and discussed the impact of this relationship on model comparison. Celeux et al [58] have further investigated the problem with DIC based on the conditional likelihood for the latent variable models and have discussed various alternatives.…”

Bayes factors: Prior sensitivity and model generalizability

“…Choosing the model focus, they argued, depends on whether the researcher wishes to make future predictions for the upper-level units or for the lowerlevel units. This issue is currently an active area of research in Bayesian statistics (Celeux, Forbes, Robert, & Titterington, 2006;Trevisani & Gelfand, 2003). We do not wish to claim that one level of analysis is more appropriate than than another, but rather that the local-global distinction -following and -is one that deserves wider recognition in the mathematical psychology literature.…”

Bayesian Comparison of Latent Variable Models: Conditional Versus Marginal Likelihoods

“…There are additional reasons to generally prefer the marginal likelihood, including the incidental parameter problem (Neyman & Scott, 1948;Lancaster, 2000), of which a Bayesian version exists when Bayes modal inference is used (also see Mislevy, 1986;O'Hagan, 1976). Spiegelhalter et al (2002) discuss the conditional/marginal distinction in their original paper on the DIC, where they refer to the issue as "model focus" (see also Celeux, Forbes, Robert, Titterington, et al, 2006;Millar, 2009;Trevisani & Gelfand, 2003). If the parameters "in focus" include the latent variables, the likelihood becomes what we call the conditional likelihood; otherwise the likelihood is what we call the marginal likelihood.…”

“…Appendix A Posterior expectations of marginal and conditional likelihoods Following Trevisani and Gelfand (2003), who studied DIC in the context of linear mixed models, we can use Jensen's inequality to show that the posterior expected value of the marginal log-likelihood is less than the posterior expected value of the conditional loglikelihood.…”