Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm

Abstract

Valiant (1984) and others have studied the problem of learning various classes of Boolean functions from examples. Here we discuss incremental learning of these functions. We consider a setting in which the learner responds to each example according to a current hypothesis. Then the learner updates the hypothesis, if necessary, based on the correct classification of the example. One natural measure of the quality of learning in this setting is the number of mistakes the learner makes. For suitable classes of functions, learning algorithms are available that make a bounded number of mistakes, with the bound independent of the number of examples seen by the learner. We present one such algorithm that learns disjunctive Boolean functions, along with variants for learning other classes of Boolean functions. The basic method can be expressed as a linear-threshold algorithm. A primary advantage of this algorithm is that the number of mistakes grows only logarithmically with the number of irrelevant attributes in the examples. At the same time, the algorithm is computationally efficient in both time and space.

Approximating the Online Set Multicover Problems via Randomized Winnowing

“…The goal is to minimize the total cost of the selected sets 1 . In this paper, we describe a new randomized algorithm for the online multicover problem based on a randomized version of the winnowing approach of [15]. This algorithm generalizes and improves some earlier results in [1,2].…”

Sequence Mining in Categorical Domains: Algorithms and Applications

“…To evaluate the effectiveness of FeatureMine, we used the feature set it produces as input to two standard classification algorithms: Winnow (Littlestone, 1988) and Naive Bayes (Duda and Hart, 1973). We ran experiments on three datasets described below.…”

Language and Automata Theory and Applications

“…It is well known that in a variety of passive learning models, such as Valiant's PAC model, [58], and the mistake bound models of Littlestone and Haussler et al, [47,38], it is intractable to learn or even approximate classical automata, [34,4,56]. However, the problem becomes tractable when the learner is allowed to make membership and equivalence queries, as in the active model of learning introduced by Angluin, [4,5].…”