Consistency of Support Vector Machines and Other Regularized Kernel Classifiers

Steinwart, Ingo

doi:10.1109/tit.2004.839514

Cited by 164 publications

(164 citation statements)

References 37 publications

Supporting

Mentioning

159

Contrasting

Unclassified

Order By: Relevance

“…3, the regularization parameter λ = (m ∧ n) −0.9 guarantees the statistical consistency of KuLSIF and KL-div under mild assumptions. Statistical properties of KLR have been studied by Bartlett et al (2006), Bartlett and Tewari (2007), Steinwart (2005), and Park (2009). Especially, Steinwart (2005) has proved that, under mild assumptions, KLR with λ = (m+n) −0.9 has the statistical consistency.…”

Section: Statistical Performancementioning

confidence: 99%

“…Statistical properties of KLR have been studied by Bartlett et al (2006), Bartlett and Tewari (2007), Steinwart (2005), and Park (2009). Especially, Steinwart (2005) has proved that, under mild assumptions, KLR with λ = (m+n) −0.9 has the statistical consistency. When the training samples are balanced, i.e., the ratio of sample size m/n converges to a positive constant, the regularization parameter λ = (m ∧ n) −0.9 guarantees the statistical consistency of KLR.…”

Section: Statistical Performancementioning

confidence: 99%

See 1 more Smart Citation

Statistical analysis of kernel-based least-squares density-ratio estimation

2011

View full text Add to dashboard Cite

The ratio of two probability densities can be used for solving various machine learning tasks such as covariate shift adaptation (importance sampling), outlier detection (likelihood-ratio test), feature selection (mutual information), and conditional probability estimation. Several methods of directly estimating the density ratio have recently been developed, e.g., moment matching estimation, maximum-likelihood density-ratio estimation, and least-squares density-ratio fitting. In this paper, we propose a kernelized variant of the least-squares method for density-ratio estimation, which is called kernel unconstrained leastsquares importance fitting (KuLSIF). We investigate its fundamental statistical properties including a non-parametric convergence rate, an analytic-form solution, and a leave-oneout cross-validation score. We further study its relation to other kernel-based density-ratio estimators. In experiments, we numerically compare various kernel-based density-ratio estimation methods, and show that KuLSIF compares favorably with other approaches.

show abstract

Section: Statistical Performancementioning

confidence: 99%

Section: Statistical Performancementioning

confidence: 99%

Statistical analysis of kernel-based least-squares density-ratio estimation

2011

View full text Add to dashboard Cite

show abstract

“…This handles all of the cases shown in Figure 1 except the support vector machine. Steinwart (2002) has demonstrated consistency for the support vector machine as well, in a general setting where F is taken to be a reproducing kernel Hilbert space, and φ is assumed continuous. Other results on Bayes-risk consistency have been presented by Breiman (2000), Jiang (2003), Mannor and Meir (2001), and Mannor et al (2002).…”

Section: Introductionmentioning

confidence: 98%

Convexity, Classification, and Risk Bounds

Bartlett

Jordan

McAuliffe

2006

Journal of the American Statistical Association

755

907

View full text Add to dashboard Cite

Many of the classification algorithms developed in the machine learning literature, including the support vector machine and boosting, can be viewed as minimum contrast methods that minimize a convex surrogate of the 0-1 loss function. The convexity makes these algorithms computationally efficient. The use of a surrogate, however, has statistical consequences that must be balanced against the computational virtues of convexity. To study these issues, we provide a general quantitative relationship between the risk as assessed using the 0-1 loss and the risk as assessed using any nonnegative surrogate loss function. We show that this relationship gives nontrivial upper bounds on excess risk under the weakest possible condition on the loss function: that it satisfy a pointwise form of Fisher consistency for classification. The relationship is based on a simple variational transformation of the loss function that is easy to compute in many applications. We also present a refined version of this result in the case of low noise. Finally, we present applications of our results to the estimation of convergence rates in the general setting of function classes that are scaled convex hulls of a finite-dimensional base class, with a variety of commonly used loss functions.

show abstract

“…They showed that if the start and end vertex kernels have the so-called universality property (e.g. Gaussian kernel) [32], then the resulting Kronecker edge kernel is also universal, resulting in universal consistency when training kernel-based learning algorithms, such as support vector machines and ridge regression using the kernel.…”

Section: Related Workmentioning

confidence: 99%

Fast Kronecker Product Kernel Methods via Generalized Vec Trick

Airola

Pahikkala

2018

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

Kronecker product kernel provides the standard approach in the kernel methods literature for learning from graph data, where edges are labeled and both start and end vertices have their own feature representations. The methods allow generalization to such new edges, whose start and end vertices do not appear in the training data, a setting known as zero-shot or zero-data learning. Such a setting occurs in numerous applications, including drug-target interaction prediction, collaborative filtering and information retrieval. Efficient training algorithms based on the so-called vec trick, that makes use of the special structure of the Kronecker product, are known for the case where the training data is a complete bipartite graph. In this work we generalize these results to non-complete training graphs. This allows us to derive a general framework for training Kronecker product kernel methods, as specific examples we implement Kronecker ridge regression and support vector machine algorithms. Experimental results demonstrate that the proposed approach leads to accurate models, while allowing order of magnitude improvements in training and prediction time.

show abstract

Consistency of Support Vector Machines and Other Regularized Kernel Classifiers

Cited by 164 publications

References 37 publications

Statistical analysis of kernel-based least-squares density-ratio estimation

Statistical analysis of kernel-based least-squares density-ratio estimation

Convexity, Classification, and Risk Bounds

Fast Kronecker Product Kernel Methods via Generalized Vec Trick

Contact Info

Product

Resources

About