Performance thresholding in practical text classification

Abstract

In practical classification, there is often a mix of learnable and unlearnable classes and only a classifier above a minimum performance threshold can be deployed. This problem is exacerbated if the training set is created by active learning. The bias of actively learned training sets makes it hard to determine whether a class has been learned. We give evidence that there is no general and efficient method for reducing the bias and correctly identifying classes that have been learned. However, we characterize a number of scenarios where active learning can succeed despite these difficulties.

ILAB: An Interactive Labelling Strategy for Intrusion Detection

“…This step corresponds to uncertainty sampling [20], a classical active learning method applied in [1]. Uncertainty sampling suffers, however, from sampling bias [29]. We also perform rare category detection to foster the discovery of yet unknown families.…”

“…Active learning methods have been proposed to reduce the labelling cost by asking the expert to annotate only the most informative examples [32]. However, classical active learning methods often suffer from sampling bias [29,34]: a family (a group of similar malicious or benign examples) may be completely overlooked by the annotation queries as the expert is asked to annotate only the most informative examples. Sampling bias is a significant issue in intrusion detection: it may lead to missing a malicious family during the labelling process, and being unable to detect it thereafter.…”

“…Active learning methods rely on an interactive process where the expert is asked to annotate some instances from a large unlabelled pool to improve the current detection model and the relevance of the future annotation queries (see Figure 1). However, annotating only the most informative instances may cause a family of observations to be completely missed by the labelling process (see [8,29] for theoretical examples) and, therefore, may have a negative impact on the performance of the detection model.…”

Iterative Constrained Clustering for Subjectivity Word Sense Disambiguation

“…During AL, as more and more labels are obtained, the training set quickly diverges from the underlying data distribution. (Schütze et al, 2006) states that AL can explore the feature space in such a biased way that it can end up ignoring entire clusters of unlabeled instances. We think that SWSD is highly prone for the mentioned missed cluster problem because of its unique nature.…”

Hierarchical sampling for active learning

“…And this is just in one dimension; in high dimension, the problem can be expected to be worse, since there are more places for this troublesome group to be hiding out. For a discussion of this problem in text classification, see the recent paper of Schutze et al (2006).…”