Approximate resilience, monotonicity, and the complexity of agnostic learning

Dachman-Soled, Dana; Feldman, Vitaly; Tan, Li-Yang; Wan, Andrew; Wimmer, Karl

doi:10.1137/1.9781611973730.34

Cited by 11 publications

(16 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More closer to our work, [1,4,7] provide positive learning results using gradient-based algorithms, but do not show the benefit of a convolutional architecture for optimization performance, compared to a fully-connected architecture. The hardness of learning in the case of Boolean functions, using the degree of the target function, was discussed in the statistical queries literature, for instance in [6]. In terms of techniques, our construction is inspired by target functions proposed in [18,19], and based on ideas from the statistical queries literature (e.g.…”

Section: Related Workmentioning

confidence: 99%

Weight Sharing is Crucial to Succesful Optimization

Shalev-Shwartz,

Shamir,

Shammah

2017

Preprint

View full text Add to dashboard Cite

Exploiting the great expressive power of Deep Neural Network architectures, relies on the ability to train them. While current theoretical work provides, mostly, results showing the hardness of this task, empirical evidence usually differs from this line, with success stories in abundance. A strong position among empirically successful architectures is captured by networks where extensive weight sharing is used, either by Convolutional or Recurrent layers. Additionally, characterizing specific aspects of different tasks, making them "harder" or "easier", is an interesting direction explored both theoretically and empirically. We consider a family of ConvNet architectures, and prove that weight sharing can be crucial, from an optimization point of view. We explore different notions of the frequency, of the target function, proving necessity of the target function having some low frequency components. This necessity is not sufficient -only with weight sharing can it be exploited, thus theoretically separating architectures using it, from others which do not. Our theoretical results are aligned with empirical experiments in an even more general setting, suggesting viability of examination of the role played by interleaving those aspects in broader families of tasks.

show abstract

Section: Related Workmentioning

confidence: 99%

Weight Sharing is Crucial to Succesful Optimization

Shalev-Shwartz,

Shamir,

Shammah

2017

Preprint

View full text Add to dashboard Cite

show abstract

“…This is implicit in prior work [Kalai et al, 2008, Feldman, 2012 and we provide additional details in Section 5. Dachman-Soled et al [2015] recently showed that ℓ 1 approximation by polynomials is necessary and sufficient condition for agnostic learning over a product distribution (at least in the statistical query framework of Kearns [1998]). Our agnostic learning algorithm (Theorem 1.4) and lower bound for polynomial approximation (Theorem 1.1) demonstrate that this equivalence does not hold for non-product distributions.…”

Section: Corollary 14 There Is An Algorithm That Agnostically Learns ...mentioning

confidence: 99%

Agnostic Learning of Disjunctions on Symmetric Distributions

Feldman,

Kothari

2014

Preprint

Self Cite

View full text Add to dashboard Cite

We consider the problem of approximating and learning disjunctions (or equivalently, conjunctions) on symmetric distributions over {0, 1} n . Symmetric distributions are distributions whose PDF is invariant under any permutation of the variables. We prove that for every symmetric distribution D, there exists a set of n O(log (1/ǫ)) functions S, such that for every disjunction c, there is function p, expressible as a linear combination of functions in S, such that p ǫ-approximates c in ℓ 1 distance on D or Ex∼D[|c(x) − p(x)|] ≤ ǫ. This implies an agnostic learning algorithm for disjunctions on symmetric distributions that runs in time n O(log (1/ǫ)) . The best known previous bound is n O(1/ǫ 4 ) and follows from approximation of the more general class of halfspaces [Wimmer, 2010]. We also show that there exists a symmetric distribution D, such that the minimum degree of a polynomial that 1/3-approximates the disjunction of all n variables in ℓ 1 distance on D is Ω( √ n). Therefore the learning result above cannot be achieved via ℓ 1 -regression with a polynomial basis used in most other agnostic learning algorithms.Our technique also gives a simple proof that for any product distribution D and every disjunction c, there exists a polynomial p of degree O(log (1/ǫ)) such that p ǫ-approximates c in ℓ 1 distance on D. This was first proved by Blais et al. [2008] via a more involved argument.

show abstract

“…For example, if the unknown function is Boolean then learning with ℓ 1 error is equivalent to learning with Boolean disagreement error [KKMS08]. In fact, it is known that the complexity of agnostic learning over product distributions in the statistical query model is characterized by how well the Boolean functions can be approximated in ℓ 1 by low-degree polynomials [DSFTWW15]. Applications of learning algorithms for submodular functions to differentially-private data release require ℓ 1 error [GHRU11; CKKL12; FK14] as does learning of probabilistic concepts (which are concepts expressing the probability of an event) [KS94].…”

Section: Introductionmentioning

confidence: 99%