Feature Subset Selection for Ordered Logit Model via Tangent-Plane-Based Approximation

Abstract

This paper is concerned with a mixed-integer optimization (MIO) approach to selecting a subset of relevant features from among many candidates. For ordinal classification, a sequential logit model and an ordered logit model are often employed. For feature subset selection in the sequential logit model, Sato et al. [22] recently proposed a mixed-integer linear optimization (MILO) formulation. In their MILO formulation, a univariate nonlinear function contained in the sequential logit model was represented by a tangent-line-based approximation. We extend this MILO formulation toward the ordered logit model, which is more commonly used for ordinal classification than the sequential logit model is. Making use of tangent planes to approximate a bivariate nonlinear function involved in the ordered logit model, we derive an MILO formulation for feature subset selection in the ordered logit model. Our computational results verify that the proposed method is superior to the L1-regularized ordered logit model in terms of solution quality.

Stochastic Discrete First-Order Algorithm for Feature Subset Selection

“…Another direction of future research is to conduct theoretical analyses of the algorithm's performance. We are now working on extending our algorithm to mixed-integer optimization models for ordinal regression and classification [4], [19], [20], [25], [28], [29].…”

Sparse Poisson regression via mixed-integer optimization

“…This algorithm adds tangent lines one by one so that the total approximation gap (the area of the shaded portion in Fig 2 ) will be minimized. Naganuma et al [ 38 ] employed a greedy algorithm that selects tangent planes to approximate the bivariate nonlinear function for ordinal classification. This algorithm iteratively selects tangent points where the approximation gap is largest.…”

“…Optimization software can solve the resultant MILO problems to optimality. Greedy algorithms for selecting a limited number of linear functions for piecewise-linear approximations have also been developed [ 38 , 40 ].…”

“…In contrast to many sparse estimation algorithms, the MIO approach has the advantage of finding the best subset of explanatory variables with respect to various criterion functions, including Mallows' C p [30], adjusted R 2 [31], information criteria [31][32][33], mRMR [34], and the cross-validation criterion [35]. MIO-based sparse estimation methods can be extended to binary or ordinal classification models [36][37][38][39][40] and to eliminating multicollinearity [41][42][43][44].…”

“…The log-likelihood to be maximized is a concave but nonlinear function, making it hard to apply an MIO approach to sparse Poisson regression. To remedy such nonlinearity, prior studies made effective use of piecewise-linear approximations of the log-likelihood functions, thereby yielding mixed-integer linear optimization (MILO) formulations for binary or ordinal classification [38][39][40]. Optimization software can solve the resultant MILO problems to optimality.…”