Sharp Oracle Inequalities for Stationary Points of Nonconvex Penalized M-Estimators

Elsener, Andreas; Geer, Sara van de

doi:10.1109/tit.2018.2863700

Cited by 11 publications

(7 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The estimation errors match the lower bounds up to constant factor using Tukey's depth (Mizera et al 2002). Elsener and van de Geer (2018) and Loh (2017) studied non-convex M-estimators. Lugosi and Mendelson (2019) and Lugosi and Mendelson (2020) considered the median-of-mean tournament (Jerrum, Valiant andVazirani 1986, Nemirovsky andYudin 1983).…”

Section: Introductionmentioning

confidence: 52%

Adversarial robust weighted Huber regression

Sasai

Fujisawa

2021

Preprint

View full text Add to dashboard Cite

We propose a novel method to estimate the coefficients of linear regression when outputs and inputs are contaminated by malicious outliers. Our method consists of two-step: (i) Make appropriate weights { ŵi } n i=1 such that the weighted sample mean of regression covariates robustly estimates the population mean of the regression covariate, (ii) Process Huber regression using { ŵi } n i=1 . When (a) the regression covariate is a sequence with i.i.d. random vectors drawn from sub-Gaussian distribution with unknown mean and known identity covariance and (b) the absolute moment of the random noise is finite, our method attains a faster convergence rate than Diakonikolas, Kong and Stewart (2019) and Cherapanamjeri et al. (2020). Furthermore, our result is minimax optimal up to constant factor. When (a) the regression covariate is a sequence with i.i.d. random vectors drawn from heavy tailed distribution with unknown mean and bounded kurtosis and (b) the absolute moment of the random noise is finite, our method attains a convergence rate, which is minimax optimal up to constant factor.MSC 2010 subject classifications: 62G35, 62G05.

show abstract

Section: Introductionmentioning

confidence: 52%

Adversarial robust weighted Huber regression

Sasai

Fujisawa

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Remark We make use of the same terminology as in Elsener and van de Geer () for condition : empirical process condition. To verify the empirical process condition, we often need ad hoc techniques.…”

Section: Deterministic Resultsmentioning

confidence: 99%

“…As far as the nonconvex estimator is concerned, it is convenient to view it as a penalized empirical risk minimizer. The empirical risk and its theoretical counterpart, the risk, are defined for all β as

R_{n} (β) = \frac{1}{4} {‖\tilde{normal∑} - β β^{T}‖}_{F}^{2}, R (β) = E R_{n} (β) .

Their derivatives are respectively given by

{\dot{R}}_{n} (β) = \frac{\partial}{\partial β} R_{n} (β), \dot{R} (β) = E {\dot{R}}_{n} (β) .

The estimator

\overset{β}{true}

was proposed in van de Geer () and further analysed in Janková and van de Geer () and Elsener and van de Geer () in the special case of sparse PCA. The following lemma parallels Lemma 2 of Janková and van de Geer ().…”

Section: Methodsmentioning

confidence: 99%

Sparse spectral estimation with missing and corrupted measurements

Elsener

Geer

2019

Stat

Self Cite

View full text Add to dashboard Cite

Supervised learning methods with missing data have been extensively studied not just due to the techniques related to low‐rank matrix completion. Also, in unsupervised learning, one often relies on imputation methods. As a matter of fact, missing values induce a bias in various estimators such as the sample covariance matrix. In the present paper, a convex method for sparse subspace estimation is extended to the case of missing and corrupted measurements. This is done by correcting the bias instead of imputing the missing values. The estimator is then used as an initial value for a nonconvex procedure to improve the overall statistical performance. The methodological and theoretical frameworks are applied to a wide range of statistical problems. These include sparse principal component analysis with different types of randomly missing data. Finally, the statistical performance is demonstrated on synthetic data.

show abstract

“…Non-convex optimization and statistics: Lastly, an active line of work exist in nonconvex estimation problems in which various statistical and algorithmic guarantees of a non-convex M-estimator are studied (Loh and Wainwright, 2012;Yang et al, 2015;Mei et al, 2018;Elsener and van de Geer, 2019). Our objective turns out to be a non-convex function of parameters, and our work utilizes a number of tools in non-convex literature to obtain statistical and algorithmic guarantees of the proposed estimator which is a stationary point of the non-convex objective function.…”

Section: Related Workmentioning

confidence: 99%

Prediction in the presence of response-dependent missing labels

Song

Raskutti

Willett

2021

Preprint

View full text Add to dashboard Cite

In a variety of settings, limitations of sensing technologies or other sampling mechanisms result in missing labels, where the likelihood of a missing label in the training set is an unknown function of the data. For example, satellites used to detect forest fires cannot sense fires below a certain size threshold. In such cases, training datasets consist of positive and pseudo-negative observations where pseudo-negative observations can be either true negatives or undetected positives with small magnitudes. We develop a new methodology and non-convex algorithm P(ositive) U(nlabeled) -O(ccurrence) M(agnitude) M(ixture) which jointly estimates the occurrence and detection likelihood of positive samples, utilizing prior knowledge of the detection mechanism. Our approach uses ideas from positive-unlabeled (PU)-learning and zero-inflated models that jointly estimate the magnitude and occurrence of events. We provide conditions under which our model is identifiable and prove that even though our approach leads to a nonconvex objective, any local minimizer has optimal statistical error (up to a log term) and projected gradient descent has geometric convergence rates. We demonstrate on both synthetic data and a California wildfire dataset that our method out-performs existing state-of-the-art approaches.

show abstract

Sharp Oracle Inequalities for Stationary Points of Nonconvex Penalized M-Estimators

Cited by 11 publications

References 34 publications

Adversarial robust weighted Huber regression

Adversarial robust weighted Huber regression

Sparse spectral estimation with missing and corrupted measurements

Prediction in the presence of response-dependent missing labels

Contact Info

Product

Resources

About