Sparse Multinomial Logistic Regression via Approximate Message Passing

Abstract

Abstract-For the problem of multi-class linear classification and feature selection, we propose approximate message passing approaches to sparse multinomial logistic regression (MLR). First, we propose two algorithms based on the Hybrid Generalized Approximate Message Passing (HyGAMP) framework: one finds the maximum a posteriori (MAP) linear classifier and the other finds an approximation of the test-error-rate minimizing linear classifier. Then we design computationally simplified variants of these two algorithms. Next, we detail methods to tune the hyperparameters of their assumed statistical models using Stein's unbiased risk estimate (SURE) and expectationmaximization (EM), respectively. Finally, using both synthetic and real-world datasets, we demonstrate improved error-rate and runtime performance relative to existing state-of-the-art approaches to sparse MLR.

Performance Limits With Additive Error Metrics in Noisy Multimeasurement Vector Problems

“…Simulation results demonstrated the promise of our algorithm. APPENDIX A. Derivation of g out for logistic channels (7) Byrne and Schniter [29] provide a method to derive g out for logistic channels (7), but the actual formula for g out is not given in their paper. To make our paper self-contained, we outline the derivation of g out for logistic channels.…”

“…For logistic channels (7), [40], where u max is the maximum number of Gaussian CDF's one wants to use, Φ( w σu/a ) denotes the Gaussian CDF whose standard deviation is σu a , and α u is the weight. Following Byrne and Schniter [29], we define the i-th…”

“…Apart from g out , we also need to find the partial derivative of g out (29), which according to Rangan [21] [13]. Suppose that we have already obtained the MMSE for an MMV problem.…”

“…3 For the special case of i.i.d. joint Bernoulli-Gaussian signals where φ(x n ) ∼ N (0, I) in (2) and I is an identity matrix, f an and f vn in Lines 18 and 19 are given below, 3 Byrne and Schniter [29] describe without detail how to derive gout(·) (4) for i.i.d. parallel logistic channels (7); we present a detailed derivation for completeness in Appendix A, and do not claim it as a contribution.…”

“…Therefore, it is difficult to calculate E[w|k, y, Θ]. Instead of calculating E[w|k, y, Θ] by brute force, Byrne and Schniter[29] use a mixture of Guassian cumulative distribution functions (CDF's) to approximate the…”