The complexity of properly learning simple concept classes

Abstract

We consider the complexity of properly learning concept classes, i.e. when the learner must output a hypothesis of the same form as the unknown concept. We present the following new upper and lower bounds on well-known concept classes:• We show that unless NP = RP, there is no polynomial-time PAC learning algorithm for DNF formulas where the hypothesis is an OR-of-thresholds. Note that as special cases, we show that neither DNF nor OR-of-thresholds are properly learnable unless NP = RP. Previous hardness results have required strong restrictions on the size of the output DNF formula. We also prove that it is NP-hard to learn the intersection of 2 halfspaces by the intersection of k halfspaces for any constant k 0. Previous work held for the case when k = .• Assuming that NP DTIME(2 n ) for a certain constant < 1 we show that it is not possible to learn size s decision trees by size s k decision trees for any k 0. Previous hardness results for learning decision trees held for k 2. • We present the first non-trivial upper bounds on properly learning DNF formulas. More specifically, we show how to learn size s DNF by DNF in time 2Õ ( √ n log s) .The hardness results for DNF formulas and intersections of halfspaces are obtained via specialized graph products for amplifying the hardness of approximating the chromatic number as well as applying recent work on the hardness of approximate hypergraph coloring. The hardness results for decision trees, as well as the new upper bounds, are obtained by developing a connection between automatizability in proof complexity and learnability, which may have other applications.

Hardness of approximate two-level logic minimization and PAC learning with membership queries

“…Valiant's original definition required that the learning algorithm output a DNF expression but this restriction was later relaxed to any efficiently-evaluatable hypothesis with the stricter version being referred to as proper learning. All these variants of the DNF learning question remained open until a recent result by Alekhnovich et al that established NP-hardness of the hardest variant: proper learning from random examples only [2]. Building on their proof, we resolve one more of Valiant's questions: Theorem 1.2 (Informal).…”

“…These results were strengthened by Nock et al [29] who proved similar hardness even when learning by formulas of size k α n β (where α 2 and β is any constant). Finally, Alekhnovich et al removed any bounds on the size of the hypothesis (other than those naturally imposed by the polynomial running time of the learning algorithm) [2]. Angluin and Kharitonov prove that if non-uniform one-way functions exist then MQs do not help predicting DNF formulae [5].…”

On the hardness of learning intersections of two halfspaces

“…However, a result such as Theorem 1 was not known for learning intersections of (two) halfspaces. Blum and Rivest [8] showed that it is NP-hard to learn an intersection of two halfspaces with an intersection of two halfspaces, and Alekhnovich, Braverman, Feldman, Klivans and Pitassi [1] proved a similar result even when the hypothesis is an intersection of halfspaces for any constant . Both the results are only NP-hardness results and do not prove APX-hardness for the underlying optimization problem.…”

“…Note that the imperfect completeness is necessary in the above theorem, since via linear programming, one can always efficiently find a halfspace that correctly classifies all the points, if one exists. The theorem is optimal, since one can easily classify 1 2 fraction of the data points correctly, by taking an arbitrary halfspace or its complement as a hypothesis. From the learning theory perspective, such an optimal hardness result is especially satisfying, since if one could efficiently find ( 1 2 + ε)-consistent hypothesis (i.e.…”

On Noise-Tolerant Learning of Sparse Parities and Related Problems

“…While there has been significant recent progress on the problem, including that random DNF are learnable under the uniform distribution [12,24,25], virtually nothing is known about their learnability in the worst case, even in the classical noiseless PAC model. In fact, there are serious impediments to learning DNF, including hardness results for their proper learnability [1]. The previous best algorithm for learning s-term DNF from random examples is due to Verbeurgt [27] and runs in quasipolynomial time O(n log s ), where is the error rate.…”