We consider the problem of estimating the mean and covariance of a distribution from iid samples in R n , in the presence of an η fraction of malicious noise; this is in contrast to much recent work where the noise itself is assumed to be from a distribution of known type. The agnostic problem includes many interesting special cases, e.g., learning the parameters of a single Gaussian (or finding the best-fit Gaussian) when η fraction of data is adversarially corrupted, agnostically learning a mixture of Gaussians, agnostic ICA, etc. We present polynomial-time algorithms to estimate the mean and covariance with error guarantees in terms of informationtheoretic lower bounds. As a corollary, we also obtain an agnostic algorithm for Singular Value Decomposition. * Georgia Tech.

We provide improved convergence rates for constrained convex-concave min-max problems and monotone variational inequalities with higher-order smoothness. In min-max settings where the p th -order derivatives are Lipschitz continuous, we give an algorithm HigherOrderMirrorProx that achieves an iteration complexity of O(1/T p+1 2 ) when given access to an oracle for finding a fixed point of a p th -order equation. We give analogous rates for the weak monotone variational inequality problem. For p > 2, our results improve upon the iteration complexity of the first-order Mirror Prox method of Nemirovski [2004] and the second-order method of Monteiro and Svaiter [2012]. We further instantiate our entire algorithm in the unconstrained p = 2 case.

No abstract

We study the design of differentially private algorithms for adaptive analysis of dynamically growing databases, where a database accumulates new data entries while the analysis is ongoing. We provide a collection of tools for machine learning and other types of data analysis that guarantee differential privacy and accuracy as the underlying databases grow arbitrarily large. We give both a general technique and a specific algorithm for adaptive analysis of dynamically growing databases. Our general technique is illustrated by two algorithms that schedule black box access to some algorithm that operates on a fixed database to generically transform private and accurate algorithms for static databases into private and accurate algorithms for dynamically growing databases. These results show that almost any private and accurate algorithm can be rerun at appropriate points of data growth with minimal loss of accuracy, even when data growth is unbounded. Our specific algorithm directly adapts the private multiplicative weights algorithm of [HR10] to the dynamic setting, maintaining the accuracy guarantee of the static setting through unbounded data growth. Along the way, we develop extensions of several other differentially private algorithms to the dynamic setting, which may be of independent interest for future work on the design of differentially private algorithms for growing databases.

No abstract

No abstract

We present preconditioned stochastic gradient descent (SGD) algorithms for the ℓ 1 minimization problem min x Ax − b 1 in the overdetermined case, where there are far more constraints than variables. Specifically, we have A ∈ R n×d for n ≫ d. Commonly known as the Least Absolute Deviations problem, ℓ 1 regression can be used to solve many important combinatorial problems, such as minimum cut and shortest path. SGD-based algorithms are appealing for their simplicity and practical efficiency. Our primary insight is that careful preprocessing can yield preconditioned matrices à with strong properties (besides good condition number and low-dimension) that allow for faster convergence of gradient descent. In particular, we precondition using Lewis weights to obtain an isotropic matrix with fewer rows and strong upper bounds on all row norms. We leverage these conditions to find a good initialization, which we use along with recent smoothing reductions and accelerated stochastic gradient descent algorithms to achieve ǫ relative error in O(nnz(A) + d 2.5 ǫ −2 ) time with high probability, where nnz(A) is the number of non-zeros in A. This improves over the previous best result using gradient descent for ℓ 1 regression. We also match the best known running times for interior point methods in several settings.Finally, we also show that if our original matrix A is approximately isotropic and the row norms are approximately equal, we can give an algorithm that avoids using fast matrix multiplication and obtains a running time of O(nnz(A)+sd 1.5 ǫ −2 +d 2 ǫ −2 ), where s is the maximum number of nonzeros in a row of A. In this setting, we beat the best interior point methods for certain parameter regimes.

No abstract