Robust Estimation via Robust Gradient Estimation

Prasad, Ashutosh; Suggala, Arun Sai; Balakrishnan, Sivaraman; Ravikumar, Pradeep

doi:10.48550/arxiv.1802.06485

Cited by 47 publications

(97 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Davis et al [7] proposed proxBoost that is based on robust distance estimation and proximal operators. Prasad et al [42] utilized the geometric median-ofmeans to robustly estimate gradients in each mini-batch. Gorbunov et al [21] and Nazin et al [39] proposed clipped-SSTM and RSMD respectively based on truncation of stochastic gradients for stochastic mirror/gradient descent.…”

Section: Related Workmentioning

confidence: 99%

“…However, all the above works [7,42,21,39] have an unfavorable O(n) dependency on the batch size to get the typical O(1/n) convergence rate (on the squared error). We note that our bound is comparable to the above approaches while using a constant batch size.…”

Section: Related Workmentioning

confidence: 99%

“…Finally, we describe why clipped-SGD allows a O(1) batch size at a high level. For previous works of stochastic optimization with strongly convex and smooth objective, they use a constant step size throughout their training process [19,42,7,39]. However, to ensure their approach make a constant progress at each iteration, they should use an exponential growing batch size to reduce variance of gradients.…”

Section: Heavy-tailed Linear Regressionmentioning

confidence: 99%

“…There has also been a burgeoning line of recent work that relaxes these assumptions and allows for heavy-tailed underlying distributions [3,28,35], but the resulting algorithms are often not only complex, but are also specifically batch learning algorithms that require the entire dataset, which limits their scalability. For instance, many popular polynomial time algorithms on heavy-tailed mean estimation [9,43,5,44,6,32] and heavy-tailed linear regression [28,46,42,38] need to store the entire dataset. On the other hand, most successful practical modern learning algorithms are iterative, light-weight and access data in a "streaming" fashion.…”

Section: Introductionmentioning

confidence: 99%

“…While there has been a line of work on heavy-tailed stochastic optimization, these require non-trivial storage complexity or batch sizes, making them unsuitable for streaming settings or large-scale problems [16,37]. Specifically these existing works require at least O(1/ ) batch size to obtain a -approximate solution under heavy-tailed noise [42,7,21,39] (See Section B in the Appendix for further discussion). In other words, to achieve a typical O(1/N ) convergence rate (on the squared error), where N is the number of samples, they would need a batch-size of nearly an entire dataset.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Heavy-tailed Streaming Statistical Estimation

Tsai¹,

Prasad²,

Balakrishnan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We consider the task of heavy-tailed statistical estimation given streaming p-dimensional samples. This could also be viewed as stochastic optimization under heavy-tailed distributions, with an additional O(p) space complexity constraint. We design a clipped stochastic gradient descent algorithm and provide an improved analysis, under a more nuanced condition on the noise of the stochastic gradients, which we show is critical when analyzing stochastic optimization problems arising from general statistical estimation problems. Our results guarantee convergence not just in expectation but with exponential concentration, and moreover does so using O(1) batch size. We provide consequences of our results for mean estimation and linear regression. Finally, we provide empirical corroboration of our results and algorithms via synthetic experiments for mean estimation and linear regression.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Heavy-tailed Linear Regressionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Heavy-tailed Streaming Statistical Estimation

Tsai¹,

Prasad²,

Balakrishnan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Excess risk bounds in robust empirical risk minimization

Minsker

Mathieu

2021

Information and Inference: A Journal of the IMA

View full text Add to dashboard Cite

This paper investigates robust versions of the general empirical risk minimization algorithm, one of the core techniques underlying modern statistical methods. Success of the empirical risk minimization is based on the fact that for a ‘well-behaved’ stochastic process $\left \{ f(X), \ f\in \mathscr F\right \}$ indexed by a class of functions $f\in \mathscr F$, averages $\frac{1}{N}\sum _{j=1}^N f(X_j)$ evaluated over a sample $X_1,\ldots ,X_N$ of i.i.d. copies of $X$ provide good approximation to the expectations $\mathbb E f(X)$, uniformly over large classes $f\in \mathscr F$. However, this might no longer be true if the marginal distributions of the process are heavy tailed or if the sample contains outliers. We propose a version of empirical risk minimization based on the idea of replacing sample averages by robust proxies of the expectations and obtain high-confidence bounds for the excess risk of resulting estimators. In particular, we show that the excess risk of robust estimators can converge to $0$ at fast rates with respect to the sample size $N$, referring to the rates faster than $N^{-1/2}$. We discuss implications of the main results to the linear and logistic regression problems and evaluate the numerical performance of proposed methods on simulated and real data.

show abstract

Robust Risk Minimization for Statistical Learning From Corrupted Data

Osama

Zachariah

Stoica

2020

IEEE Open J. Signal Process.

View full text Add to dashboard Cite

We consider a general statistical learning problem where an unknown fraction of the training data is corrupted. We develop a robust learning method that only requires specifying an upper bound on the corrupted data fraction. The method minimizes a risk function defined by a non-parametric distribution with unknown probability weights. We derive and analyse the optimal weights and show how they provide robustness against corrupted data. Furthermore, we give a computationally efficient coordinate descent algorithm to solve the risk minimization problem. We demonstrate the wide range applicability of the method, including regression, classification, unsupervised learning and classic parameter estimation, with state-of-the-art performance.

show abstract

Robust Estimation via Robust Gradient Estimation

Cited by 47 publications

References 0 publications

Heavy-tailed Streaming Statistical Estimation

Heavy-tailed Streaming Statistical Estimation

Excess risk bounds in robust empirical risk minimization

Robust Risk Minimization for Statistical Learning From Corrupted Data

Contact Info

Product

Resources

About