On the equivalence of weak learnability and linear separability: new relaxations and efficient boosting algorithms

Abstract

Boosting algorithms build highly accurate prediction mechanisms from a collection of lowaccuracy predictors. To do so, they employ the notion of weak-learnability. The starting point of this paper is a proof which shows that weak learnability is equivalent to linear separability with ℓ 1 margin. While this equivalence is a direct consequence of von Neumann's minimax theorem, we derive the equivalence directly using Fenchel duality. We then use our derivation to describe a family of relaxations to the weak-learnability assumption that readily translates to a family of relaxations of linear separability with margin. This alternative perspective sheds new light on known soft-margin boosting algorithms and also enables us to derive several new relaxations of the notion of linear separability. Last, we describe and analyze an efficient boosting framework that can be used for minimizing the loss functions derived from our family of relaxations. In particular, we obtain efficient boosting algorithms for maximizing hard and soft versions of the ℓ 1 margin.

Trading Accuracy for Sparsity in Optimization Problems with Sparsity Constraints

“…The proof uses ideas from convex analysis. We refer the reader to [3,23] and see also a similar derivation in [26]. The definition ofL implies that it is the infimal convolution of L and the quadratic function (β/2)v 2 .…”

Parallel Boosting with Momentum

“…As before, we will choose a linear functionL t ≤ L in each round and squash φ t towards it to obtain the new auxiliary function. Therefore (14) continues to hold, and we can again inductively prove that φ t continues to retain an elliptical quadratic form:…”

“…However, parallel boosting algorithms on their own are too slow. See for instance [14] for a primal-dual analysis of the rate of convergence of boosting algorithms in the context of loss minimization.…”

A robust semi-supervised boosting method using linear programming

“…n ′ is the regularization parameter which denotes the level of relaxation of the margin constraint. Shalev-Shwartz and Singer [16] proved that optimizing the above linear programming problem is equivalent to maximizing the averaged n ′ smallest margins. Thus, the case where n ′ equals 1 denotes maximizing the minimum margin, the case where n ′ equal n means maximizing the averaged margin over all labeled examples, and n ′ lies in the range from 1 to n.…”

“…SSLPBoost actually maximizes a combination of the averaged n ′ smallest margins over the labeled data and averaged m ′ smallest margins over the unlabeled data. ShalevShwartz [16] showed the use of soft margins with supervised learning. Here, we only consider the margins for the unlabeled data.…”