Agnostic Learning of Monomials by Halfspaces Is Hard

Abstract

We prove the following strong hardness result for learning: Given a distribution of labeled examples from the hypercube such that there exists a monomial consistent with (1 − ǫ) of the examples, it is NP-hard to find a halfspace that is correct on (1/2 + ǫ) of the examples, for arbitrary constants ǫ > 0. In learning theory terms, weak agnostic learning of monomials is hard, even if one is allowed to output a hypothesis from the much bigger concept class of halfspaces. This hardness result subsumes a long line of previous results, including two recent hardness results for the proper learning of monomials and halfspaces. As an immediate corollary of our result we show that weak agnostic learning of decision lists is NP-hard.Our techniques are quite different from previous hardness proofs for learning. We define distributions on positive and negative examples for monomials whose first few moments match. We use the invariance principle to argue that regular halfspaces (all of whose coefficients have small absolute value relative to the total ℓ 2 norm) cannot distinguish between distributions whose first few moments match. For highly non-regular subspaces, we use a structural lemma from recent work on fooling halfspaces to argue that they are "junta-like" and one can zero out all but the top few coefficients without affecting the performance of the halfspace. The top few coefficients form the natural list decoding of a halfspace in the context of dictatorship tests/Label Cover reductions.We note that unlike previous invariance principle based proofs which are only known to give Unique-Games hardness, we are able to reduce from a version of Label Cover problem that is known to be NP-hard. This has inspired follow-up work on bypassing the Unique Games conjecture in some optimal geometric inapproximability results.Before describing the details of the prior body of work on hardness results for learning, we note that our result subsumes all these results with just one exception (the hardness of learning monomials by t- CNFs [34]). This is because we obtain the optimal inapproximability factor and allow learning of monomials by the much richer class of halfspaces.The results of the paper are noteworthy in the broader context of hardness of approximation. Previously, hardness proofs based on the invariance principle were only known to give Unique-Games

Tikhonov, Ivanov and Morozov regularization for support vector machine learning

“…On the contrary, the use of a nonconvex loss leads to NP-problems, which cannot be exactly solved for sample sets whose cardinality exceeds few tens of data (e.g. n > 30) (Anthony 2001;Feldman et al 2009), but for which approximate solutions can be eventually found (Lawler and Wood 1966;Yuille and Rangarajan 2003). As a matter of fact, if one has to cope with a non-convex loss, a convex relaxation is often used in order to reformulate the problem so to make it computationally tractable.…”

Demonstrating non-inferiority of easy interpretable methods for insolvency prediction

“…Typically, the α-error is regarded much worse than the β-error. Unfortunately, empirical risk minimization in the case of the hypothesis class of DNFs is NP-hard because this is already the case for the subclass of monomials (Feldman et al, 2012). Thus, heuristics come into play.…”

Bypassing UGC from some Optimal Geometric Inapproximability Results

“…It was first introduced in [21] for proving hardness results in hypergraph coloring and subsequently utilized for other applications in [25] [26] and [16].…”

“…A key ingredient in our reductions is the smooth version of Label Cover which enables us to devise a more sophisticated decoding procedure which can be combined with the dictatorship test. It is pertinent to note that a couple of the (few) previous results using smooth versions of Label Cover, on hardness of learning intersection of halfspaces [26] and monomials [16], have also used analysis based on versions of the Central Limit Theorem. This work suggests the possibility of obtaining optimal NP-hardness results for other combinatorial optimization problems using reductions based on smooth versions of Label Cover.…”