Discussion: Bayesian regularization and effect smoothing for categorical predictors

Abstract

IntroductionWe would first like to thank Gerhard Tutz and Jan Gertheiss for their profound review of regularization methods for categorical variables, either as covariates or as response variables in regression models. Categorical variables are rather the rule than the exception in regression analyses, particularly in the medical, social and economic sciences. It is amazing that it took quite a long time from ridge regression (Hoerl and Kennard, 1970) to versions of regularization methods, which take into account the specific structure of categorical covariates. As Tutz and Gertheiss predominantly discuss penalized estimation, we will focus on the Bayesian perspective to regularization and effect fusion for categorical covariates.Regularization and sparsity in regression type models have been addressed from a Bayesian point of view in the literature, see for example, Fahrmeir et al. (2010) for an overview on Bayesian regularization and George and McCulloch (1995), Ishwaran and Rao (2005) and O'Hara and Sillanpäa (2009) for approaches to Bayesian variable selection. However, the specific issues arising for categorical covariates have not yet received much attention, with the exception of Chipman (1996), who considers a Bayesian approach for groupwise selection of level effects of a categorical covariate (the Bayesian pendant to Section 3.2.1 of Tutz and Gertheiss, 2016) and the recently proposed Bayesian pendant to smoothing predictors and effect fusion based on spike and slab priors in Pauger and Wagner (2016).Here, we will discuss the relation between regularization penalties and prior distributions and then describe a Bayesian approach for smoothing of level effects of categorical covariates. Finally, we will briefly describe spike and slab prior distributions which encourage a sparse representation of their effects.

Regularization approaches in clinical biostatistics: A review of methods and their applications

“…40 Also in the Bayesian framework, fusion of effects has been investigated by several researchers, see, e.g., Pauger et al 41 or Malsiner-Walli et al 42 A nice discussion on Bayesian regularization and effect smoothing for categorical predictors can be found in Wagner and Pauger. 43 Semiparametric function estimation with smoothness priors where a flexible effect f false( x false) of a covariate x of interest shall be estimated. One option is to work with function spaces and associated norms such as the functional L2 loss pen 0.2em false( .1em .1em f false) = λ ∫ false( .1em .1em f ″ false( x false) ) 2 d x, i.e., the integrated squared second derivative that penalizes the curvature of the function.…”

“…40 Also in the Bayesian framework, fusion of effects has been investigated by several researchers, see, e.g., Pauger et al 41 or Malsiner-Walli et al 42 A nice discussion on Bayesian regularization and effect smoothing for categorical predictors can be found in Wagner and Pauger. 43…”