Penalized Bregman divergence for large-dimensional regression and classification

Abstract

Regularization methods are characterized by loss functions measuring data fits and penalty terms constraining model parameters. The commonly used quadratic loss is not suitable for classification with binary responses, whereas the loglikelihood function is not readily applicable to models where the exact distribution of observations is unknown or not fully specified. We introduce the penalized Bregman divergence by replacing the negative loglikelihood in the conventional penalized likelihood with Bregman divergence, which encompasses many commonly used loss functions in the regression analysis, classification procedures and machine learning literature. We investigate new statistical properties of the resulting class of estimators with the number p(n) of parameters either diverging with the sample size n or even nearly comparable with n, and develop statistical inference tools. It is shown that the resulting penalized estimator, combined with appropriate penalties, achieves the same oracle property as the penalized likelihood estimator, but asymptotically does not rely on the complete specification of the underlying distribution. Furthermore, the choice of loss function in the penalized classifiers has an asymptotically relatively negligible impact on classification performance. We illustrate the proposed method for quasilikelihood regression and binary classification with simulation evaluation and real-data application.

High-dimensional regression and classification under a class of convex loss functions

“…However, they are particularly interested in quantile regression and gaussian graphical modeling respectively. Recently, to generalize the conventional penalized likelihood, Zhang et al (2010) introduced penalized Bregman divergence. It used the concept of Bregman divergence, which unifies nearly all of the commonly used loss functions in the regression analysis and classification procedure (Zhang et al, 2009).…”

“…For instance, an important application of Bregman divergence is the quasi-likelihood model (Wedderburn, 1974) which is popular when the underlying distribution of the observations is not fully specified. Zhang et al (2010) studied the statistical properties of the penalized Bregman divergence estimator in conjunction with either nonconvex or convex penalties. The dimension p n in that work has either a smaller or nearly the same order as the sample size n, depending on the choice of penalties.…”

Regularized estimation of hemodynamic response function for fMRI data

“…Areas for future research include (I) more efficient methods for estimating the large error covariance matrix, (II) more rigorous investigation of the sampling properties of penalized estimators under the convolution model, (III) confidence intervals for h n in spirit similar to that of Sara et al (2004) and hypothesis testing of h n = 0 for detecting activation via the penalized estimators, and (IV) related multiple comparison procedures and activation maps. It's worthy mentioning that Zhang, Jiang and Chai (2008) developed statistical inference tools for testing the significance of large-dimensional parameters via penalized estimators, when the data are independent and identically distributed. More refined work is needed for generalizing those inference methods to the fMRI time series, which are serially correlated.…”

“…To quantify the error measures for different types of response variables, Zhang, Jiang and Chai (2008) considered a broad class of loss functions Q(·, ·), called Bregman divergence (BD). The penalized-BD estimator ( β n,0 , β n ) is defined as the minimizer of the following criterion function,…”

“…Under the class of Bregman divergence as the general loss functions, Zhang, Jiang and Chai (2008) studied the consistency rate, sparsity property and asymptotic normality of the penalized-BD estimator in the relatively lowdimensional case,…”