Compressed Kernel Perceptrons

Abstract

Kernel machines are a popular class of machine learning algorithms that achieve state of the art accuracies on many real-life classification problems. Kernel perceptrons are among the most popular online kernel machines that are known to achieve high-quality classification despite their simplicity. They are represented by a set of B prototype examples, called support vectors, and their associated weights. To obtain a classification, a new example is compared to the support vectors. Both space to store a prediction model and time to provide a single classification scale as O(B). A problem with kernel perceptrons is that the number of support vectors tends to grow without bounds with the number of training examples on noisy data. To reduce the strain at computational resources, budget kernel perceptrons have been developed by upper bounding the number of support vectors. In this work, we proposed a new budget algorithm that upper bounds the number of bits needed to store kernel perceptron. Setting the bitlength constraint could facilitate development of hardware and software implementations of kernel perceptrons on resource-limited devices such as microcontrollers. The proposed compressed kernel perceptron algorithm decides on the optimal tradeoff between number of support vectors and their bit precision. The algorithm was evaluated on several benchmark data sets and the results indicate that it can train highly accurate classifiers even when the available memory budget drops below 1 Kbit. This promising result points to a possibility of implementing powerful learning algorithms even on the most resourceconstrained computational devices.

Prototype-based budget maintenance for tracking in depth videos

“…The general strategy for adding a support vector is to add it at its first appearance, while the decision to remove it is more critical and arbitrary. Therefore, several approaches have been proposed for removal: [7,28] remove samples based on a random selection; [9] removes the oldest support vectors; while [8,29,30] and also Struck [12] remove the support vector that causes the minimum L 2 norm change to the SVM primal model. In this work, we have decided to follow a different approach based on the notion of prototype selection [21].…”

Online training on a budget of support vector machines using twin prototypes

“…To maintain the budget, the support vector 4 , which would be predicted most confidently, is removed when the budget is exceeded. The related strategies include removing a random SV4,7, the oldest SV6, and the SV that would results in the smallest prediction error8,43. Projection is another popular budget maintenance strategy and has been widely adopted by many online kernel algorithms44–46.…”

“…Development of a similarly successful algorithm for nonlinear regression is still an open problem. Representatives of hybrid approaches in classification are budgeted kernel perceptrons4–8. The kernel perceptrons are represented by a subset of observed examples (i.e., support vectors) and their weights, and the budgeted solution is achieved by ensuring that the number of support vectors is bounded.…”

Support Vector Number Reduction: Survey and Experimental Evaluations

“…The method in [65] starts from an empty set and iteratively adds an SV that is expected to make the largest decrease in the approximation error. Reversely, in [67] and [68], they start from the original SV set and iteratively remove an SV that is expected to make the least increase in the approximation error. In particular, in [68], an efficient method is proposed for the calculation of the approximation error of the reduced SV set, which should be iteratively recalculated while SVs are being removed, by exploiting the kernel matrix of the original SV set.…”