General convergence results for linear discriminant updates

Abstract

Abstract. The problem of learning linear-discriminant concepts can be solved by various mistake-driven update procedures, including the Winnow family of algorithms and the well-known Perceptron algorithm. In this paper we define the general class of "quasi-additive" algorithms, which includes Perceptron and Winnow as special cases. We give a single proof of convergence that covers a broad subset of algorithms in this class, including both Perceptron and Winnow, but also many new algorithms. Our proof hinges on analyzing a generic measure of progress construction that gives insight as to when and how such algorithms converge.Our measure of progress construction also permits us to obtain good mistake bounds for individual algorithms. We apply our unified analysis to new algorithms as well as existing algorithms. When applied to known algorithms, our method "automatically" produces close variants of existing proofs (recovering similar bounds)-thus showing that, in a certain sense, these seemingly diverse results are fundamentally isomorphic. However, we also demonstrate that the unifying principles are more broadly applicable, and analyze a new class of algorithms that smoothly interpolate between the additive-update behavior of Perceptron and the multiplicative-update behavior of Winnow.

Untitled

“…In work parallel to this, the general additive update (9) in the context of linear classification, i.e., with a thresholded transfer function, has recently been developed and analyzed by Grove, Littlestone, and Schuurmans (1997) with methods and results very similar to ours; see also Gentile and Littlestone (1999). Gentile and Warmuth (1999) have shown how the notion of matching loss can be generalized to thresholded transfer functions.…”

“…The normalized Winnow algorithm (Littlestone, 1989) is analogous to the exponentiated gradient algorithm: the loss bound depends on the product of the ∞-norm of the instances and the 1-norm of the correct weight vector. Grove, Littlestone, and Schuurmans (1997) have generalized this by giving for each p > 2 a classification algorithm that has a loss bound in terms of on the p-norm of the instances and the q-norm of the correct weight vector, where 1/ p + 1/q = 1. This result for general norm pairs has also been extended to linear regression (Gentile & Littlestone, 1999).…”

A primal-dual perspective of online learning algorithms

“…Online learning of linear classifiers is an important and well-studied domain in machine learning with interesting theoretical properties and practical applications (Cesa-Bianchi et al 2002;Crammer et al 2005;Gentile 2001Grove et al 2001;Helmbold et al 1999;Kivinen et al 2002;Kivinen and Warmuth 1997;Li and Long 2002). An online learning algorithm observes instances in a sequence of trials.…”

“…The flurry of online learning algorithms sparked unified analyses of seemingly different online algorithms by Littlestone, Warmuth, Kivinen and colleagues (Kivinen and Warmuth 1997;Littlestone 1988). Most notably is the work of Grove, Littlestone, and Schuurmans (Grove et al 2001) on a quasi-additive family of algorithms, which includes both the Perceptron (Rosenblatt 1958) and the Winnow (Littlestone 1988) algorithms as special cases. A similar unified view for regression was derived by Warmuth (1997, 2001).…”

“…A similar unified view for regression was derived by Warmuth (1997, 2001). Online algorithms for linear hypotheses and their analyses became more general and powerful by employing Bregman divergences for measuring the progress toward a good hypothesis Grove et al 2001;Kivinen et al 2002).…”