We propose a Bayesian approach to obtain a sparse representation of the effect of a categorical predictor in regression type models. As this effect is captured by a group of level effects, sparsity cannot only be achieved by excluding single irrelevant level effects or the whole group of effects associated to this predictor but also by fusing levels which have essentially the same effect on the response. To achieve this goal, we propose a prior which allows for almost perfect as well as almost zero dependence between level effects a priori. This prior can alternatively be obtained by specifying spike and slab prior distributions on all effect differences associated to this categorical predictor. We show how restricted fusion can be implemented and develop an efficient MCMC method for posterior computation. The performance of the proposed method is investigated on simulated data and we illustrate its application on real data from EU-SILC.keywords: spike and slab prior, sparsity, nominal and ordinal predictor, regression model, MCMC, Gibbs sampler
Abstract:In social and economic studies many of the collected variables are measured on a nominal scale, often with a large number of categories. The definition of categories can be ambiguous and different classification schemes using either a finer or a coarser grid are possible. Categorization has an impact when such a variable is included as covariate in a regression model: a too fine grid will result in imprecise estimates of the corresponding effects, whereas with a too coarse grid important effects will be missed, resulting in biased effect estimates and poor predictive performance.To achieve an automatic grouping of the levels of a categorical covariate with essentially the same effect, we adopt a Bayesian approach and specify the prior on the level effects as a location mixture of spiky Normal components. Model-based clustering of the effects during MCMC sampling allows to simultaneously detect categories which have essentially the same effect size and identify variables with no effect at all. Fusion of level effects is induced by a prior on the mixture weights which encourages empty components. The properties of this approach are investigated in simulation studies. Finally, the method is applied to analyse effects of high-dimensional categorical predictors on income in Austria.
IntroductionWe would first like to thank Gerhard Tutz and Jan Gertheiss for their profound review of regularization methods for categorical variables, either as covariates or as response variables in regression models. Categorical variables are rather the rule than the exception in regression analyses, particularly in the medical, social and economic sciences. It is amazing that it took quite a long time from ridge regression (Hoerl and Kennard, 1970) to versions of regularization methods, which take into account the specific structure of categorical covariates. As Tutz and Gertheiss predominantly discuss penalized estimation, we will focus on the Bayesian perspective to regularization and effect fusion for categorical covariates.Regularization and sparsity in regression type models have been addressed from a Bayesian point of view in the literature, see for example, Fahrmeir et al. (2010) for an overview on Bayesian regularization and George and McCulloch (1995), Ishwaran and Rao (2005) and O'Hara and Sillanpäa (2009) for approaches to Bayesian variable selection. However, the specific issues arising for categorical covariates have not yet received much attention, with the exception of Chipman (1996), who considers a Bayesian approach for groupwise selection of level effects of a categorical covariate (the Bayesian pendant to Section 3.2.1 of Tutz and Gertheiss, 2016) and the recently proposed Bayesian pendant to smoothing predictors and effect fusion based on spike and slab priors in Pauger and Wagner (2016).Here, we will discuss the relation between regularization penalties and prior distributions and then describe a Bayesian approach for smoothing of level effects of categorical covariates. Finally, we will briefly describe spike and slab prior distributions which encourage a sparse representation of their effects.
No abstract
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.