An empirical study of the behavior of active learning for word sense disambiguation

Chen, Jinying; Schein, Andrew I.; Ungar, Lyle H.; Palmer, Martha

doi:10.3115/1220835.1220851

Cited by 46 publications

(35 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A trend of the last ten years (Abe and Mamitsuka 1998;Banko and Brill 2001;Chen et al 2006;Dagan and Engelson 1995;Hwa 2004;Lewis and Gale 1994;McCallum and Nigam 1998;Melville and Mooney 2004;Roy and McCallum 2001;Tang et al 2002) has been to employ heuristic methods of active learning with no explicitly defined objective function. Uncertainty sampling (Lewis and Gale 1994), query by committee (Seung et al 1992), 1 and variants have proven particularly attractive because of their portability across a wide spectrum of machine learning algorithms.…”

Section: Background and Related Workmentioning

confidence: 99%

Active learning for logistic regression: an evaluation

2007

View full text Add to dashboard Cite

Which active learning methods can we expect to yield good performance in learning binary and multi-category logistic regression classifiers? Addressing this question is a natural first step in providing robust solutions for active learning across a wide variety of exponential models including maximum entropy, generalized linear, log-linear, and conditional random field models. For the logistic regression model we re-derive the variance reduction method known in experimental design circles as 'A-optimality.' We then run comparisons against different variations of the most widely used heuristic schemes: query by committee and uncertainty sampling, to discover which methods work best for different classes of problems and why. We find that among the strategies tested, the experimental design methods are most likely to match or beat a random sample baseline. The heuristic alternatives produced mixed results, with an uncertainty sampling variant called margin sampling and a derivative method called QBB-MM providing the most promising performance at very low computational cost. Computational running times of the experimental design methods were a bottleneck to the evaluations. Meanwhile, evaluation of the heuristic methods lead to an accumulation of negative results. We explore alternative evaluation design parameters to test whether these negative results are merely an artifact of settings where experimental design methods can be applied. The results demonstrate a need for improved active learning methods that will provide reliable performance at a reasonable computational cost.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Active learning for logistic regression: an evaluation

2007

View full text Add to dashboard Cite

show abstract

“…In this article, we are interested in uncertainty sampling schemes [Lewis and Gale 1994] for pool-based active learning, which in recent years has been widely studied in tasks such as word sense disambiguation [Chen et al 2006;Chan and Ng 2007], Text Classification (TC) [Lewis and Gale 1994;Zhu et al 2008b], statistical syntactic parsing [Tang et al 2002], and named entity recognition [Shen et al 2004].…”

Section: Active Learning Processmentioning

confidence: 99%

“…The main difference among the various pool-based active learning algorithms is the method of assessing the uncertainty of each unlabeled example in the pool. In the case of probabilistic models, the uncertainty of the classifier is commonly estimated using the entropy of its output [Tang et al 2002;Chen et al 2006;Zhu and Hovy 2007]. For active learning with nonprobabilistic models such as support vector machines [Tong and Koller 2001;Schohn and Cohn 2000], the classification margin is used.…”

Section: Active Learning Processmentioning

confidence: 99%

“…Committee-based sampling [Seung et al 1992;Dagan and Engelson 1995;McCallum and Nigam 1998a] generates a committee of classifiers and selects the next unlabeled example by the principle of maximal disagreement among these classifiers, which is uncertainty sampling with an ensemble. By using these selective sampling techniques, the size of the labeled training data can be significantly reduced for text classification [Lewis and Gale 1994;McCallum and Nigam 1998a] and word sense disambiguation [Chen et al 2006;Zhu and Hovy 2007].…”

Section: Introductionmentioning

confidence: 99%

“…Active learning aims to minimize the amount of human labeling effort required for a supervised classifier to achieve a satisfactory performance [Cohn et al 1996]. In recent years active learning has been widely studied in various Natural Language Processing (NLP) tasks, such as Word Sense Disambiguation (WSD) [Chen et al 2006;Zhu and Hovy 2007;Chan and Ng 2007], Text Classification (TC) [Lewis and Gale 1994;McCallum and Nigam 1998a;Tong and Koller 2001], Named Entity Recognition (NER) [Shen et al 2004;Jones 2005;Tomanek et al 2007], chunking [Ngai and Yarowsky 2000], Information Extraction (IE) [Thompson et al 1999;Culotta and McCallum 2005], and statistical parsing [Hwa 2000;Tang et al 2002;Becker and Osborne 2005].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Confidence-based stopping criteria for active learning for data annotation

Zhu

Wang

Hovy

et al. 2010

ACM Trans. Speech Lang. Process.

View full text Add to dashboard Cite

The labor-intensive task of labeling data is a serious bottleneck for many supervised learning approaches for natural language processing applications. Active learning aims to reduce the human labeling cost for supervised learning methods. Determining when to stop the active learning process is a very important practical issue in real-world applications. This article addresses the stopping criterion issue of active learning, and presents four simple stopping criteria based on confidence estimation over the unlabeled data pool, including maximum uncertainty, overall uncertainty, selected accuracy, and minimum expected error methods. Further, to obtain a proper threshold for a stopping criterion in a specific task, this article presents a strategy by considering the label change factor to dynamically update the predefined threshold of a stopping criterion during the active learning process. To empirically analyze the effectiveness of each stopping criterion for active learning, we design several comparison experiments on seven real-world datasets for three representative natural language processing applications such as word sense disambiguation, text classification and opinion analysis.

show abstract

Linguistic Annotation

Palmer

Xue

2010

The Handbook of Computational Linguistics and Natural Language Processing

Self Cite

View full text Add to dashboard Cite

Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The publisher is not associated with any product or vendor mentioned in this book. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought.

show abstract

An empirical study of the behavior of active learning for word sense disambiguation

Cited by 46 publications

References 8 publications

Active learning for logistic regression: an evaluation

Active learning for logistic regression: an evaluation

Confidence-based stopping criteria for active learning for data annotation

Linguistic Annotation

Contact Info

Product

Resources

About