General Bayesian Loss Function Selection and the use of Improper Models

Abstract

Statisticians often face the choice between using probability models or a paradigm defined by minimising a loss function. Both approaches are useful and, if the loss can be re-cast into a proper probability model, there are many tools to decide which model or loss is more appropriate for the observed data, in the sense of explaining the data's nature. However, when the loss leads to an improper model, there are no principled ways to guide this choice. We address this task by combining the Hyvärinen score, which naturally targets infinitesimal relative probabilities, and general Bayesian updating, which provides a unifying framework for inference on losses and models. Specifically we propose the H-score, a general Bayesian selection criterion and prove that it consistently selects the (possibly improper) model closest to the data-generating truth in Fisher's divergence.We also prove that an associated H-posterior consistently learns optimal hyper-parameters featuring in loss functions, including a challenging tempering parameter in generalised Bayesian inference.As salient examples, we consider robust regression and non-parametric density estimation where popular loss functions define improper models for the data and hence cannot be dealt with using standard model selection tools. These examples illustrate advantages in robustness-efficiency tradeoffs and provide a Bayesian implementation for kernel density estimation, opening a new avenue for Bayesian non-parametrics.

On Selection Criteria for the Tuning Parameter in Robust Divergence

“…as an unnormalized statistical model whose normalizing constant may not exist. Recently, Jewson and Rossell (2021) pointed out that the role of such unnormalized models can be recognized in terms of relative probability. For such model, we employ the Hyvarinen score (H-score) in terms of Bayesian model selection (Shao et al, 2019;Dawid and Musio, 2015), defined as…”

“…On the other hand, Jewson and Rossell (2021) proposed an information criterion via Laplace approximation of the marginal likelihood in which the potential function is constructed by the Hyvarinen score. Although Jewson and Rossell (2021) covers unnormalized models with possibly diverging normalizing constants, the estimator used in the criterion is entirely different from one defined as the maximizer of robust divergence; thereby, the criterion does not apply to the tuning parameter selection of robust divergence either. Moreover, Yonekura and Sugasawa (2021) developed an robust sequential Monte Carlo sampler based on robust divergence in which γ is adaptively selected.…”

Adaptation of the Tuning Parameter in General Bayesian Inference with Robust Divergence

“…Recently, Jewson and Rossell (2021) has introduced a new Bayesian framework called H-posterior for unnormalized statistical models based on Fisher divergence, and they developed model selection criterion via the Laplace approximation of the marginal likelihood.…”

“…The biggest difference from Jewson and Rossell (2021) is that we use a natural form of general posterior based on robust divergence, which is widely adopted in literature (e.g. Jewson et al, 2018;Ghosh and Basu, 2016), while the form of the posterior distribution in Jewson and Rossell (2021) is different from ours.…”

“…The biggest difference from Jewson and Rossell (2021) is that we use a natural form of general posterior based on robust divergence, which is widely adopted in literature (e.g. Jewson et al, 2018;Ghosh and Basu, 2016), while the form of the posterior distribution in Jewson and Rossell (2021) is different from ours. Moreover, we also provide a tailored numerical algorithm using an SMC sampler that enables us to automatically select the tuning parameter through sampling iterations.…”