Local Rademacher Complexity: Sharper risk bounds with and without unlabeled samples

Oneto, Luca; Ghio, Alessandro; Ridella, Sandro; Anguita, Davide

doi:10.1016/j.neunet.2015.02.006

Cited by 33 publications

(49 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, we will show the three main results in the context of the complexity‐based methods: The first approaches deal with the problem of finite‐sized sets of rules: the Union Bound method (Bonferroni, ; Vapnik, ), which takes into account the whole set of rules, and the shell bound method (Langford & McAllester, , ), which takes into account just the rules with small empirical error; The second approaches are based on the seminal work of V. N. Vapnik and A. Chernovenkis and deal with infinite‐sized sets of rules for the particular case of binary classification: the VC theory (Vapnik, ), which takes into account the whole set of rules, and the local VC theory (Oneto, Anguita et al, ) which takes into account just the rules with small empirical error. Extensions to the general SL framework have been proposed over the years (Bartlett, Kulkarni, & Posner, ; Shawe‐Taylor et al, ; Vapnik, ; Zhou, ), but were overly complicated and eventually made obsolete by the Rademacher complexity theory; The last approach is the Rademacher complexity theory which deals with infinite‐sized sets of rules and the general SL framework: the global Rademacher complexity theory (Bartlett & Mendelson, ; Koltchinskii, ; Oneto et al, ; Oneto, Ghio et al, ), which takes into account the whole set of rules, and the local Rademacher complexity theory (Bartlett et al, ; Bartlett, Bousquet, & Mendelson, ; Koltchinskii, ; Lugosi & Wegkamp, ; Oneto, Ghio et al, ) which takes into account just the rules with small empirical error. …”

Section: Complexity‐based Methodsmentioning

confidence: 99%

“…As an example, data‐dependent versions of the VC theory have been developed in Boucheron et al () and Shawe‐Taylor et al () studies. In recent years, researchers have also succeeded in developing local data‐dependent complexity measures (Bartlett et al, ; Bartlett, Bousquet, & Mendelson, ; Blanchard & Massart, ; Cortes, Kloft, & Mohri, ; Koltchinskii, ; Oneto, Anguita et al, ; Oneto, Ghio et al, ). Local measures improve over global ones thanks to their ability of taking into account only those rules of the rules class that will be most likely chosen by the learning procedure, that is, the models with small error.…”

Section: Complexity‐based Methodsmentioning

confidence: 99%

“…For example, RC has been exploited in the transductive (El‐Yaniv & Pechyony, ) and semisupervised (Katrenko & Van Zaanen, ; Kumar, Saha, & Daume, ; Rosenberg & Bartlett, ; Sun & Shawe‐Taylor, ; Zhu & Goldberg, ) learning frameworks. In the latter setting, in particular, previous works showed how the tightness of RC bounds can remarkably benefit from the exploitation of unlabeled samples (Anguita et al, ; Ben‐David, Lu, & Pál, ; Kääriäinen, ; Oneto, Ghio et al, , ).…”

Section: Complexity‐based Methodsmentioning

confidence: 99%

See 2 more Smart Citations

Model selection and error estimation without the agonizing pain

Oneto

2018

WIREs Data Min & Knowl

Self Cite

View full text Add to dashboard Cite

How can we select the best performing data-driven model? How can we rigorously estimate its generalization error? Statistical learning theory (SLT) answers these questions by deriving nonasymptotic bounds on the generalization error of a model or, in other words, by delivering upper bounding of the true error of the learned model based just on quantities computed on the available data. However, for a long time, SLT has been considered only as an abstract theoretical framework, useful for inspiring new learning approaches, but with limited applicability to practical problems. The purpose of this review is to give an intelligible overview of the problems of model selection (MS) and error estimation (EE), by focusing on the ideas behind the different SLT-based approaches and simplifying most of the technical aspects with the purpose of making them more accessible and usable in practice. We start by presenting the seminal works of the 80s until the most recent results, then discuss open problems and finally outline future directions of this field of research.

show abstract

Section: Complexity‐based Methodsmentioning

confidence: 99%

Section: Complexity‐based Methodsmentioning

confidence: 99%

Section: Complexity‐based Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Model selection and error estimation without the agonizing pain

Oneto

2018

WIREs Data Min & Knowl

Self Cite

View full text Add to dashboard Cite

show abstract

“…The classification errors yielded with each part are then averaged, and the parameterisation with the least classification error is selected [52]. Although there are studies developing methods to determine these parameters (e.g., [53,54]), the cross-validation method is still the method adopted by the majority of data analysts [51,52].…”

Section: Free-parameter Tuningmentioning

confidence: 99%

Specific Land Cover Class Mapping by Semi-Supervised Weighted Support Vector Machines

2017

View full text Add to dashboard Cite

Abstract:In many remote sensing projects on land cover mapping, the interest is often in a sub-set of classes presented in the study area. Conventional multi-class classification may lead to a considerable training effort and to the underestimation of the classes of interest. On the other hand, one-class classifiers require much less training, but may overestimate the real extension of the class of interest. This paper illustrates the combined use of cost-sensitive and semi-supervised learning to overcome these difficulties. This method utilises a manually-collected set of pixels of the class of interest and a random sample of pixels, keeping the training effort low. Each data point is then weighted according to its distance to its near positive data point to inform the learning algorithm. The proposed approach was compared with a conventional multi-class classifier, a one-class classifier, and a semi-supervised classifier in the discrimination of high-mangrove in Saloum estuary, Senegal, from Landsat imagery. The derived classification accuracies were high: 93.90% for the multi-class supervised classifier, 90.75% for the semi-supervised classifier, 88.75% for the one-class classifier, and 93.75% for the proposed method. The results show that accuracy achieved with the proposed method is statistically non-inferior to that achieved with standard binary classification, requiring however much less training effort.

show abstract

“…We demonstrate the power of our complexity bounds by applying them to derive effective generalization error bounds. from the true errors simultaneously over the whole class, while the quantity of primary importance is only that deviation for the particular function picked by the learning algorithm, which may be far from reaching this supremum [2,16,26]. Therefore, the analysis based on a global complexity would give a rather conservative estimate.…”

mentioning

confidence: 99%

Local Rademacher complexity bounds based on covering numbers

Lei

Ding

Bi³

2016

Neurocomputing

View full text Add to dashboard Cite

This paper provides a general result on controlling local Rademacher complexities, which captures in an elegant form to relate the complexities with constraint on the expected norm to the corresponding ones with constraint on the empirical norm. This result is convenient to apply in real applications and could yield refined local Rademacher complexity bounds for function classes satisfying general entropy conditions. We demonstrate the power of our complexity bounds by applying them to derive effective generalization error bounds. from the true errors simultaneously over the whole class, while the quantity of primary importance is only that deviation for the particular function picked by the learning algorithm, which may be far from reaching this supremum [2,16,26]. Therefore, the analysis based on a global complexity would give a rather conservative estimate. On the other hand, most learning algorithms are inclined towards choosing functions possessing small empirical errors and hopefully also small generalization errors [5]. Furthermore, if there holds a relationship between variances and expectations like Var(f ) ≤ B(P f ) α , these functions will also admit small variances. That is to say, the obtained prediction rule is likely to fall into a subclass with small variances [2]. Due to the seminar work of Koltchinskii and Panchenko [16] and Massart [22], it turns out that the notion of Rademacher complexity can be naturally modified to take this into account, yielding the so-called local Rademacher complexity [16]. Since local Rademacher complexity is always smaller than the global counterpart, the discussion based on local Rademacher complexities always yields significantly better learning rates under the variance-expectation conditions.Mendelson [23,24] initiated the discussion of estimating local Rademacher complexities with covering numbers and these complexity bounds are very effective in establishing fast learning rates. However, the discussions in [23,24] are somewhat dispersed in the sense that the author did not provided a general result applicable to all function classes. Indeed, Mendelson [23, 24] derived local Rademacher complexity bounds for several function classes satisfying different entropy conditions case-by-case, and the involved deduction also relies on the specific entropy conditions. Mendelson [25] also derived, for a general Reproducing Kernel Hilbert Space (RKHS), an interesting local Rademacher complexity bound based on the eigenvalues of the associated integral operator, which was later generalized to ℓ p -norm MKL context [12,13,21]. These results are exclusively developed for RKHSs and it still remains unknown whether they could be extended to general function classes. In this paper, we try to refine these discussions by providing some general and sharp results on controlling local Rademacher complexities by covering numbers. A distinguished property of our result is that it captures in an elegant form to relate local Rademacher complexities to the associated empirical local Rademach...

show abstract

Local Rademacher Complexity: Sharper risk bounds with and without unlabeled samples

Cited by 33 publications

References 21 publications

Model selection and error estimation without the agonizing pain

Model selection and error estimation without the agonizing pain

Specific Land Cover Class Mapping by Semi-Supervised Weighted Support Vector Machines

Local Rademacher complexity bounds based on covering numbers

Contact Info

Product

Resources

About