Robust estimation of <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" altimg="si1.gif" display="inline" overflow="scroll"><mml:mi>U</mml:mi></mml:math>-statistics

Joly, Émilien; Lugosi, Gábor

doi:10.1016/j.spa.2016.04.021

Cited by 34 publications

(18 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also note that, as the name indicates, the U-statistics are unbiased estimators of the kernel mean on the population of X and have minimal variance among all its unbiased estimators (see e.g. Joly and Lugosi [2016]). For the above type of kernel concerning arbitrary machine learning algorithms, ( 6) is known as leave-pair-out cross-validation (LPOCV) (see Airola et al [2011], Montoya Perez et al [2018).…”

Section: Null Hypothesis Of Label Independencementioning

confidence: 99%

“…As an extreme case of re-sampling, we can consider what we refer to as the leave-pair-out cross-validation (LPOCV), in which every differently labeled pair is held out from the sample at a time. LPOCV is an unbiased estimator of the pairwise error probability considered in this paper for any learning algorithm and it is also the lowest variance estimator of all unbiased estimators of that quantity (this follows from the theory of U-statistics for which we refer to Joly and Lugosi [2016]).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Link between Coding Theory and Cross-Validation with Applications

Pahikkala,

Movahedi,

Montoya

et al. 2021

Preprint

View full text Add to dashboard Cite

We study the combinatorics of cross-validation based AUC estimation under the null hypothesis that the binary class labels are exchangeable, that is, the data are randomly assigned into two classes given a fixed class proportion. In particular, we study how the estimators based on leave-pair-out cross-validation (LPOCV), in which every possible pair of data with different class labels is held out from the training set at a time, behave under the null without any prior assumptions of the learning algorithm or the data. It is shown that the maximal number of different fixed proportion label assignments on a sample of data, for which a learning algorithm can achieve zero LPOCV error, is the maximal size of a constant weight error correcting code, whose length is the sample size, weight is the number of data labeled with one, and the Hamming distance between code words is four. We then introduce the concept of a light constant weight code and show similar results for nonzero LPOCV errors. We also prove both upper and lower bounds on the maximal sizes of the light constant weight codes that are similar to the classical results for contant weight codes. These results pave the way towards the design of new LPOCV based statistical tests for the learning algorithms ability of distinguishing two classes from each other that are analogous to the classical Wilcoxon-Mann-Whitney U test for fixed functions. Behavior of some representative examples of learning algorithms and data are simulated in an experimental case study.

show abstract

Section: Null Hypothesis Of Label Independencementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A Link between Coding Theory and Cross-Validation with Applications

Pahikkala,

Movahedi,

Montoya

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…An intermediate step, see W MoU , is to look after diagonal blocks only as represented in (c) of the Figure 1. This formulation is also used by Lerasle et al (2019) for deriving robust mean embedding and Maximum Mean Discrepancy estimators, in order to simplify theoretical analysis due to the independency of blocks but leading to an increasing variance of the estimator (Joly and Lugosi, 2016).…”

Section: Mom and Mou-based Estimatorsmentioning

confidence: 99%

“…The Median-of-Means (MoM in short) is a robust mean estimator firstly introduced in complexity theory during the 1980s (Nemirovsky and Yudin, 1983;Jerrum et al, 1986;Alon et al, 1999). Following the seminal deviation study by Catoni (2012), MoM has recently witnessed a surge of interest, mainly due to its nice sub-gaussian behavior under the sole requirement that the second order moment is finite (Devroye et al, 2016).Originally devoted to scalar random variables, MoM has notably been extended to random vectors (Minsker et al, 2015;Hsu and Sabato, 2016;Lugosi and Mendelson, 2017) and U -statistics (Joly and Lugosi, 2016;Laforgue et al, 2019) with minimal loss of performance. As a valuable alternative to the empirical mean in presence of outliers or heavy-tailed distributions, MoM is now the cornerstone of many robust learning procedures such as bandits (Bubeck et al, 2013), robust mean embedding (Lerasle et al, 2019), or the more general frameworks of MoM-minimization (Lecué et al, 2018).…”

Section: Introductionmentioning

confidence: 99%

When OT meets MoM: Robust estimation of Wasserstein Distance

Staerman¹,

Laforgue²,

Mozharovskyi³

et al. 2020

Preprint

View full text Add to dashboard Cite

Issued from Optimal Transport, the Wasserstein distance has gained importance in Machine Learning due to its appealing geometrical properties and the increasing availability of efficient approximations. In this work, we consider the problem of estimating the Wasserstein distance between two probability distributions when observations are polluted by outliers. To that end, we investigate how to leverage Medians of Means (MoM) estimators to robustify the estimation of Wasserstein distance. Exploiting the dual Kantorovitch formulation of Wasserstein distance, we introduce and discuss novel MoM-based robust estimators whose consistency is studied under a data contamination model and for which convergence rates are provided. These MoM estimators enable to make Wasserstein Generative Adversarial Network (WGAN) robust to outliers, as witnessed by an empirical study on two benchmarks CIFAR10 and Fashion MNIST. Eventually, we discuss how to combine MoM with the entropy-regularized approximation of the Wasserstein distance and propose a simple MoM-based re-weighting scheme that could be used in conjunction with the Sinkhorn algorithm.

show abstract

“…, X i m ) we cannot expect to get exponential inequalities anymore. Nevertheless working with kernels that have finite p-th moment for some p ∈ (1, 2], Joly and Lugosi in Joly and Lugosi (2016) construct an estimator of the mean of the U-process using the median-of-means technique that performs as well as the classical U-statistic with bounded kernels.…”

Section: Introductionmentioning

confidence: 99%

Concentration inequality for U-statistics of order two for uniformly ergodic Markov chains

Duchemin¹,

Castro²,

Lacour³

2020

Preprint

View full text Add to dashboard Cite

We prove a new concentration inequality for U-statistics of order two for uniformly ergodic Markov chains. Working with bounded π-canonical kernels, we show that we can recover the convergence rate of Arcones and Gine (1993) who proved a concentration result for U-statistics of independent random variables and canonical kernels. Our proof relies on an inductive analysis where we use martingale techniques, uniform ergodicity, Nummelin splitting and Bernstein's type inequality where the spectral gap of the chain emerges.Our result allows us to conduct three applications. First, we establish a new exponential inequality for the estimation of spectra of trace class integral operators with MCMC methods. The novelty is that this result holds for kernels with positive and negative eigenvalues, which is new as far as we know.In addition, we investigate generalization performance of online algorithms working with pairwise loss functions and Markov chain samples. We provide an online-to-batch conversion result by showing how we can extract a low risk hypothesis from the sequence of hypotheses generated by any online learner.We finally give a non-asymptotic analysis of a goodness-of-fit test on the density of the invariant measure of a Markov chain. We identify the classes of alternatives over which our test based on the L 2 distance has a prescribed power.

show abstract

Robust estimation of U-statistics

Cited by 34 publications

References 14 publications

A Link between Coding Theory and Cross-Validation with Applications

A Link between Coding Theory and Cross-Validation with Applications

When OT meets MoM: Robust estimation of Wasserstein Distance

Concentration inequality for U-statistics of order two for uniformly ergodic Markov chains

Contact Info

Product

Resources

About