Fast Rates for Support Vector Machines

Steinwart, Ingo; Scovel, Clint

doi:10.1007/11503415_19

Cited by 376 publications

(661 citation statements)

References 8 publications

Supporting

Mentioning

650

Contrasting

Unclassified

Order By: Relevance

“…It has been recently shown that the coefficients a i are optimal if and only if they satisfy the set of inclusions (Steinwart 2003;De Vito et al 2004):…”

Section: Algebraic Characterization For Regularized Kernel Methodsmentioning

confidence: 99%

“…It is worth noticing that the standard approach to SVM computation relies on quadratic programming and lacks a closed-form solution. When the loss function is non-smooth, resorting to sub-differential calculus, it has been shown that the coefficients can be characterized in terms of inclusions (Steinwart 2003;De Vito et al 2004). For regression loss functions, including -insensitive SVR (Support Vector Regression), it has been recently proven that the inclusions can be converted into a set of algebraic equations by a proper change of variable (Dinuzzo et al 2007).…”

mentioning

confidence: 99%

See 1 more Smart Citation

An algebraic characterization of the optimum of regularized kernel methods

Dinuzzo

Nicolao

2009

Mach Learn

View full text Add to dashboard Cite

The representer theorem for kernel methods states that the solution of the associated variational problem can be expressed as the linear combination of a finite number of kernel functions. However, for non-smooth loss functions, the analytic characterization of the coefficients poses nontrivial problems. Standard approaches resort to constrained optimization reformulations which, in general, lack a closed-form solution. Herein, by a proper change of variable, it is shown that, for any convex loss function, the coefficients satisfy a system of algebraic equations in a fixed-point form, which may be directly obtained from the primal formulation. The algebraic characterization is specialized to regression and classification methods and the fixed-point equations are explicitly characterized for many loss functions of practical interest. The consequences of the main result are then investigated along two directions. First, the existence of an unconstrained smooth reformulation of the original non-smooth problem is proven. Second, in the context of SURE (Stein's Unbiased Risk Estimation), a general formula for the degrees of freedom of kernel regression methods is derived.

show abstract

“…It has been recently shown that the coefficients a i are optimal if and only if they satisfy the set of inclusions (Steinwart 2003;De Vito et al 2004):…”

Section: Algebraic Characterization For Regularized Kernel Methodsmentioning

confidence: 99%

mentioning

confidence: 99%

An algebraic characterization of the optimum of regularized kernel methods

Dinuzzo

Nicolao

2009

Mach Learn

View full text Add to dashboard Cite

show abstract

“…Indeed ν controls the sparsity of the solution [17], and nν is the lower bound of the number of support vectors [15]. Regarding the regularization parameter ν, the following statements hold:…”

Section: Sparsity On Example Weightsmentioning

confidence: 99%

“…3) in order to have more sparsity. Sparsity and accuracy is trade-off [17], and controlled through C and ν.…”

Section: Sparse Substructure Boosting For Regressionmentioning

confidence: 99%

Fast Iterative Mining Using Sparsity-Inducing Loss Functions

Saigo

Kashima

Tsuda³

2013

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

SUMMARYApriori-based mining algorithms enumerate frequent patterns efficiently, but the resulting large number of patterns makes it difficult to directly apply subsequent learning tasks. Recently, efficient iterative methods are proposed for mining discriminative patterns for classification and regression. These methods iteratively execute discriminative pattern mining algorithm and update example weights to emphasize on examples which received large errors in the previous iteration. In this paper, we study a family of loss functions that induces sparsity on example weights. Most of the resulting example weights become zeros, so we can eliminate those examples from discriminative pattern mining, leading to a significant decrease in search space and time. In computational experiments we compare and evaluate various loss functions in terms of the amount of sparsity induced and resulting speed-up obtained.

show abstract

“…At the same time the work elicited the problem of the growth of the testing time when a bigger training set was used, to have better recognition performances. In fact, as far as the third issue is concerned, it is well known that both the training and testing time of an SVM crucially depend on the number of samples considered [16]; as well, the number of Support Vectors (SVs) found, which determine the complexity of the solution to the problem, grows proportionally with respect to the number of samples [28]. This makes the approach unsuitable, at least so far, for on-line learning, where a potentially endless flow of data is acquired by the machine.…”

Section: Introductionmentioning

confidence: 99%

Indoor Place Recognition using Online Independent Support Vector Machines

Orabona¹,

Castellini²,

Caputo³

et al. 2007

Procedings of the British Machine Vision Conference 2007

View full text Add to dashboard Cite

In the framework of indoor mobile robotics, place recognition is a challenging task, where it is crucial that self-localization be enforced precisely, notwithstanding the changing conditions of illumination, objects being shifted around and/or people affecting the appearance of the scene. In this scenario online learning seems the main way out, thanks to the possibility of adapting to changes in a smart and flexible way. Nevertheless, standard machine learning approaches usually suffer when confronted with massive amounts of data and when asked to work online. Online learning requires a high training and testing speed, all the more in place recognition, where a continuous flow of data comes from one or more cameras. In this paper we follow the Support Vector Machines-based approach of Pronobis et al. [26], proposing an improvement that we call Online Independent Support Vector Machines. This technique exploits linear independence in the image feature space to incrementally keep the size of the learning machine remarkably small while retaining the accuracy of a standard machine. Since the training and testing time crucially depend on the size of the machine, this solves the above stated problems. Our experimental results prove the effectiveness of the approach.

show abstract

Fast Rates for Support Vector Machines

Cited by 376 publications

References 8 publications

An algebraic characterization of the optimum of regularized kernel methods

An algebraic characterization of the optimum of regularized kernel methods

Fast Iterative Mining Using Sparsity-Inducing Loss Functions

Indoor Place Recognition using Online Independent Support Vector Machines

Contact Info

Product

Resources

About