List Decodable Learning via Sum of Squares

Abstract

In the list-decodable learning setup, an overwhelming majority (say a 1 − β-fraction) of the input data consists of outliers and the goal of an algorithm is to output a small list L of hypotheses such that one of them agrees with inliers. We develop a framework for listdecodable learning via the Sum-of-Squares SDP hierarchy and demonstrate it on two basic statistical estimation problemswhere the corresponding labels y i are well-approximated by a linear function ℓ. We devise an algorithm that outputs a list L of linear functions such that there exists somel ∈ L that is close to ℓ. This yields the first algorithm for linear regression in a list-decodable setting. Our results hold for any distribution of examples whose concentration and anticoncentration can be certified by Sum-of-Squares proofs. • Mean Estimation: Given data points {X i } i∈[N] containing a subset S of βN inliers {X i } i∈S that are drawn i.i.d. from a Gaussian distribution N(µ, I) in d , we devise an algorithm that generates a list L of means such that there existsμ ∈ L close to µ. The recovery guarantees of the algorithm are analogous to the existing algorithms for the problem by Diakonikolas et al. [DKS18] and Kothari et al. [KS17a].In an independent and concurrent work, Karmalkar et al.[KKK19] also obtain an algorithm for list-decodable linear regression using the Sum-of-Squares SDP hierarchy.

Learning mixtures of linear regressions in subexponential time via Fourier moments

“…A related problem to-indeed, a generalization of-the problem of learning MLRs is that of list-decodable regression [8,26,39]. Here, we assume that we are given a set of data points (x 1 ,y 1 ), .…”

“…Unfortunately, all known techniques, including the state of the art [26,39], either are too weak to be applied to our setting, or use the Sum-of-Squares SDP hierarchy and again interact through the data via estimating high-degree moments of the distribution. As a result, these latter algorithms still suffer runtimes which are exponential in k.…”

“…At a high level, these methods require that there exists some statistic which depends only on low degree moments of the unknown distribution, so that a sufficiently good estimate of this statistic will uniquely identify the parameters of the distribution. This includes widely-used techniques based on tensor decomposition [6,40,47,48], and SDP hierarchies such as the Sum-of-Squares meta-algorithm [26,39]. If degree t moments are necessary to devise such a statistic, then these methods require exp(Ω(t )) sample and computational complexity.…”

Generalized List Decoding

“…For instance, it is used for amplifying hardness and constructing extractors, pseudorandom generators and other pseudorandom objects [20]. The idea of relaxing the problem by asking the solver to just output a list (ideally as small as possible) of solutions that is guaranteed to contain the correct one, instead of insisting on a unique answer, is also adopted in many other fields in computer science [19,35,28]. In the context of high-dimensional geometry over finite fields, list decoding is equivalent to multiple packing, just like error correction codes are equivalent to sphere packing.…”

Efficiently learning structured distributions from untrusted batches

“…We remark that the majority of this work focuses on estimation in ℓ 2 -norm or Frobenius norm, with two notable exceptions: [8] uses learning in a sparsity-inducing norm to improve the sample complexity for sparse mean estimation, and [72] gives an information-theoretic characterization of when mean estimation in general norms is possible, but they do not give efficient algorithms. Our techniques are most closely related to the sum-of-squares algorithms of [45,57], and this general technique has also found application in other robust learning problems such as robust regression [54] and list-decodable regression [52,68].…”