Improving Sparsity and Scalability in Regularized Nonconvex Truncated-Loss Learning Problems

Abstract

The truncated regular -loss support vector machine can eliminate the excessive number of support vectors (SVs); thus, it has significant advantages in robustness and scalability. However, in this paper, we discover that the associated state-of-the-art solvers, such as difference convex algorithm and concave-convex procedure, not only have limited sparsity promoting property for general truncated losses especially the -loss but also have poor scalability for large-scale problems. To circumvent these drawbacks, we present a general multistage scheme with explicit interpretation regarding SVs as well as outliers. In particular, we solve the general nonconvex truncated loss minimization through a sequence of associated convex subproblems, in which the outliers are removed in advance. The proposed algorithm can be regarded as a structural optimization attempt carefully considering sparsity imposed by the nonconvex truncated losses. We show that this general multistage algorithm offers sufficient sparsity especially for the truncated -loss. To further improve the scalability, we propose a linear multistep algorithm by employing a single iteration of coordinate descent to monotonically decrease the objective function at each stage and a kernel algorithm by using the Karush-Kuhn-Tucker conditions to cheaply find most part of the outliers for the next stage. Comparison experiments demonstrate that our methods have superiority in sparsity as well as efficiency in scalability.

Unified SVM algorithm based on LS-DC loss

“…There have been many researchers using non-convex loss function to weaken the influence of outliers. For example, Shen et al [13], Collobert et al [4], and Wu and Liu [25] study the robust SVM with the truncated hinge loss; Tao et al [20] study the robust SVM with the truncated hinge loss and the truncated squared hinge loss. Based on DCA (difference of convex algorithm) procedure [21,26], all those studies have given algorithms to iteratively solve L1/L2-SVM to obtain the solutions of their proposed non-convex models.…”

“…However, the inner loop of these algorithms is computationally expensive. For example, in Collobert et al [4], Wu and Liu [25], Feng et al [5], Tao et al [20], they solve a constrained quadratic programming (QP) defined by L1/L2-SVM or re-weighted L2-SVM, and all state-of-the-art methods for those quadratic programming require lots of iterations. In Tao et al [20], some efficient techniques based on the coordinate descent are given to reduce the cost of the inner loop, but it still needs to solve L1/L2-SVM maybe with a smaller size.…”

“…As an efficient nonconvex optimization technique, DCA, first introduced in Tao and Souad [19] and recently reviewed in Thi and Dinh [21], has been successfully applied in machine learning [4,26,20,3]…”

“…With the solution β * , all the output classification functions are f (x) = m i=1 y i β * i κ(x, x i ). To improve the robustness of model (5), the hinge loss (1 − yt) + is truncated as the ramp loss min{(1 − yt) + , a} with a > 0 and decomposed as a DC form (1 − yt) + − (1 − yt − a) + [4,25,20]; the squared hinge loss (1 − yt) 2 + is truncated as min{(1 − yt) 2 + , a} and decomposed as DC form [20]. Based on DCA, they propose algorithms to obtain the solution of the SVM models (2) with the non-convex losses by iteratively solving L1/L2-SVM.…”

“…All those algorithms for robust SVM models [4,25,20,5] need to solve a constrained QP (L1/L2-SVM or re-weight L2-SVM) in the inner loops, which result in the long training time.…”