Tracking the Best Hyperplane with a Simple Budget Perceptron

Abstract

Abstract. Shifting bounds for on-line classification algorithms ensure good performance on any sequence of examples that is well predicted by a sequence of changing classifiers. When proving shifting bounds for kernel-based classifiers, one also faces the problem of storing a number of support vectors that can grow unboundedly, unless an eviction policy is used to keep this number under control. In this paper, we show that shifting and on-line learning on a budget can be combined surprisingly well. First, we introduce and analyze a shifting Perceptron algorithm achieving the best known shifting bounds while using an unlimited budget. Second, we show that by applying to the Perceptron algorithm the simplest possible eviction policy, which discards a random support vector each time a new one comes in, we achieve a shifting bound close to the one we obtained with no budget restrictions. More importantly, we show that our randomized algorithm strikes the optimal trade-off U = Θ( √ B) between budget B and norm U of the largest classifier in the comparison sequence. Experiments are presented comparing several linear-threshold algorithms on chronologically-ordered textual datasets. These experiments support our theoretical findings in that they show to what extent randomized budget algorithms are more robust than deterministic ones when learning shifting target data streams.

Compressed Kernel Perceptrons

“…This choice was based on the simplicity of the algorithm, which is fundamental in severely resource constrained applications. Moreover, Random Perceptron achieves similar performance and has the same error bound as the more involved budget perceptrons [4]. From the memory budget viewpoint, Random Perceptron is very desirable because support vector weights are binary (they equal the class labels) and so can be stored with a single bit.…”

“…The Compressed Kernel Perceptron proposed in this paper is based on Random Perceptron [4], a budget kernel perceptron with random removal of support vectors when budget is exceeded. This choice was based on the simplicity of the algorithm, which is fundamental in severely resource constrained applications.…”

“…Given a budget of B support vectors, they achieve constant space scaling O(BM) and linear training time O(TBM). Among the proposed removal approaches are removal of a randomly selected support vector (called Random Perceptron) [4], the oldest support vector [6], or the support vector with the largest α t f(x t ) [9]. The experimental results indicate that reasonably accurate kernel perceptrons can be trained using quite modest budgets.…”

Tighter Perceptron with improved dual use of cached data for model representation and validation

“…We compared the proposed Tightest Perceptron algorithm with four state of the art budget perceptron algorithms: Self-Tuned Forgetron [6], Random Perceptron [2], and Tighter 0 and Tighter A Perceptrons [13], as well as to the baseline algorithm Stoptron where the kernel perceptron terminates once the budget is full. For Tighter A , we use A=B randomly selected examples as the additional validation set, and denote it as Tighter B .…”

Twin Vector Machines for Online Learning on a Budget

“…Development of a similarly successful algorithm for nonlinear regression is still an open problem. A representative of hybrid approaches in classification are budget kernel perceptrons [2,4,5]. The kernel perceptrons are represented by a subset of observed examples (i.e.…”