Learning juntas in the presence of noise

Abstract

The combination of two major challenges in machine learning is investigated: dealing with large amounts of irrelevant information and learning from noisy data. It is shown that large classes of Boolean concepts that depend on a small number of variables-so-called juntas-can be learned efficiently from random examples corrupted by random attribute and classification noise. To accomplish this goal, a two-phase algorithm is presented that copes with several problems arising from the presence of noise: firstly, a suitable method for approximating Fourier coefficients in the presence of noise is applied to infer the relevant variables. Secondly, as one cannot simply read off a truth table from the examples as in the noise-free case, an alternative method to build a hypothesis is established and applied to the examples restricted to the relevant variables. In particular, for the class of monotone juntas depending on d out of n variables, the sample complexity is polynomial in log(n/δ), 2 d , γ −d a , and γ −1 b , where δ is the confidence parameter and γ a , γ b > 0 are noise parameters bounding the noise rates away from 1/2. The running time is bounded by the sample complexity times a polynomial in n. So far, all results hold for the case of uniformly distributed examples-the only case that (apart from side notes) has been studied in the literature yet. We show how to extend our methods to non-uniformly distributed examples and derive new results for monotone juntas. For the attribute noise, we have to assume that it is generated by a product distribution since otherwise fault-tolerant learning is in general impossible: we construct a noise distribution P and a concept class C such that it is impossible to learn C under P-noise.

Improved Time Complexities for Learning Boolean Networks

“…Until now, there is still an open question about learning general BNs with a complexity better than n (1−o(1))(k+1) [31,32]. In this work, as an endeavor to meet the challenge, we prove that the complexity of the DFL algorithm is strictly O(k · (N + log n) · n 2 ) for learning the OR/AND BNs in the worst case, given enough noiseless random samples from the uniform distribution.…”

“…An efficient algorithm was also proposed by Mossel et al [31] with a time complexity of O(n k+1 ) ω ω+1 , which is about O(n 0.7(k+1) ), for learning arbitrary BNs, where ω < 2.376. Arpe and Reischuk [32] showed that monotonic BNs could be learned with a complexity of poly(n 2 , 2 k , log(1/δ), γ…”

Detecting controlling nodes of boolean regulatory networks

“…In fact, one can readily apply methods stemming from the area of PAC (probably approximately correct) learning theory [5], as the network identification problem can be reduced to the problem of learning Boolean juntas , i.e., Boolean functions that depend b only on a small number of their arguments. This problem was studied by Arpe and Reischuk [6] extending earlier work of Mossel et al [7,8]. …”

“…Hence, if $MathClass-rel|ĥ\left(U\right)MathClass-rel|$ is larger than 2 -k , the variables corresponding to U are classified as essential. The algorithm was given by [6], but they used 2 - d -1 as threshold (see Line 8).…”

“…The application of spectral learning algorithms leads to both better time and sample complexity for the detection of controlling nodes compared with exhaustive search. In particular, a slight improvement in the algorithm given in [6] is presented, for which analytical bounds on the number of samples needed to find the controlling nodes are derived. It is notable that for the class of 1-low networks, the time complexity of the resulting algorithms is roughly n 2 .…”