Learning general halfspaces with general Massart noise under the Gaussian distribution

Abstract

We study the task of agnostically learning halfspaces under the Gaussian distribution. Specifically, given labeled examples (x, y) from an unknown distribution on R n ×{±1}, whose marginal distribution on x is the standard Gaussian and the labels y can be arbitrary, the goal is to output a hypothesis with 0-1 loss OPT + ǫ, where OPT is the 0-1 loss of the best-fitting halfspace. We prove a near-optimal computational hardness result for this task, under the widely believed subexponential time hardness of the Learning with Errors (LWE) problem. Prior hardness results are either qualitatively suboptimal or apply to restricted families of algorithms. Our techniques extend to yield near-optimal lower bounds for related problems, including ReLU regression.

Learning a Single Neuron with Adversarial Label Noise via Gradient Descent

“…Specifically, isotropic log-concave distributions are well-behaved, i.e., they are (L, R)-well-behaved for some L, R = O(1), see, e.g., [LV07,KLT09]. Similar assumptions were introduced in [DKTZ20a] and have been used in various classification and regression settings [DKTZ20c, DKTZ20b, DKK + 20, DKK + 21, FCG20, FCG21, ZL21, ZFG21].…”

On PAC Learning Halfspaces in Non-interactive Local Privacy Model with Public Unlabeled Data

“…So, f (x) is in fact a Massart noise example oracle with λ = 3 16 . Before that we recall the definition of bounded distribution in Diakonikolas et al (2020).…”

“…Definition 14 (Bounded Distribution Diakonikolas et al (2020)) Fix U, R > 0 and t : (0, 1) → R + . An isotropic (i.e., zero mean and identity covariance) distribution…”

“…Many algorithms for computing accurate hypothesis in the distribution-specific PAC learning has been promoted, such as Awasthi et al (2015Awasthi et al ( , 2016; Zhang et al (2017). Recently, an efficient and simple algorithm has been proposed in Diakonikolas et al (2020), which succeeds under more general distributional assumptions.…”

“…In fact, several reasonable distribution families satisfy the previous two conditions. For example, the class of isotropic log-concave distribution satisfies (U, r)-anti-anti-concentration and U -anti-concentration with U, r = Θ(1) (See Fact A.4 in Diakonikolas et al (2020)). Moreover, any isotropic s-concave distribution on R d with s ≥ − 1 2d+3 satisfies (U, r)-anti-anti-concentration and U -anti-concentration with U, r = Θ(1) (See Appendix A.4 in Diakonikolas et al (2020)).…”