Learning a Single Neuron with Adversarial Label Noise via Gradient Descent

Abstract

We study the fundamental problem of learning a single neuron, i.e., a function of the form x → σ(w • x) for monotone activations σ : R → R, with respect to the L 2 2 -loss in the presence of adversarial label noise. Specifically, we are given labeled examples from a distribution D on (x, y) ∈ R d × R such that there exists w * ∈ R d achieving F (w * ) = , whereThe goal of the learner is to output a hypothesis vector w such that F ( w) = C with high probability, where C > 1 is a universal constant. As our main contribution, we give efficient constant-factor approximate learners for a broad class of distributions (including log-concave distributions) and activation functions (including ReLUs and sigmoids). Concretely, for the class of isotropic log-concave distributions, we obtain the following important corollaries: