Dual Strategy Active Learning

Dönmez, Pınar; Carbonell, Jaime G.; Bennett, Paul

doi:10.1007/978-3-540-74958-5_14

Cited by 131 publications

(105 citation statements)

References 13 publications

Supporting

Mentioning

101

Contrasting

Order By: Relevance

“…When the core of the model is consolidated, items with highest uncertainty should provide a higher improvement in performance by effectively delimiting with more precision the decision frontier of the model. This phenomenon, which lies at the heart of well-known semi-supervised learning techniques like self-training (or bootstrapping), has also been noted by approaches combining density estimation methods when very few examples are available, and uncertainty sampling when the training dataset has grown [5,17].…”

Section: Relevant Workmentioning

confidence: 82%

Information Extraction with Active Learning: A Case Study in Legal Text

Cardellino

Villata

Alemany

et al. 2015

Computational Linguistics and Intelligent Text Processing

View full text Add to dashboard Cite

Abstract. Active learning has been successfully applied to a number of NLP tasks. In this paper, we present a study on Information Extraction for natural language licenses that need to be translated to RDF. The final purpose of our work is to automatically extract from a natural language document specifying a certain license a machine-readable description of the terms of use and reuse identified in such license. This task presents some peculiarities that make it specially interesting to study: highly repetitive text, few annotated or unannotated examples available, and very fine precision needed. In this paper we compare different active learning settings for this particular application. We show that the most straightforward approach to instance selection, uncertainty sampling, does not provide a good performance in this setting, performing even worse than passive learning. Density-based methods are the usual alternative to uncertainty sampling, in contexts with very few labelled instances. We show that we can obtain a similar effect to that of density-based methods using uncertainty sampling, by just reversing the ranking criterion, and choosing the most certain instead of the most uncertain instances.

show abstract

Section: Relevant Workmentioning

confidence: 82%

Information Extraction with Active Learning: A Case Study in Legal Text

Cardellino

Villata

Alemany

et al. 2015

Computational Linguistics and Intelligent Text Processing

View full text Add to dashboard Cite

show abstract

“…This combined semi-supervised AL has the benefit of ignoring regions that can be reliably "filled in" by a semisupervised procedure, while also selecting those examples that may benefit this EM process. Donmez et al [16] propose a modification of the density-weighted technique of Nguyen and Smeulders. This modification simply selects examples according to the convex combination of the density-weighted technique and traditional uncertainty sampling.…”

Section: Density-sensitive Active Learningmentioning

confidence: 99%

“…The cold start problem has long been known to be a key difficulty in building effective classifiers quickly and cheaply via AL [13,16]. Since the quality of data selection directly depends on the understanding of the space provided by the "current" model, early stages of acquisitions can result in a vicious cycle of uninformative selections, leading to poor quality models and therefore to additional poor selections.…”

Section: Starting Coldmentioning

confidence: 99%

Class Imbalance and Active Learning

2013

Imbalanced Learning

View full text Add to dashboard Cite

Abstract:The performance of a predictive model is tightly coupled with the data used during training. While using more examples in the training will often result in a better informed, more accurate model; limits on computer memory and real-world costs associated with gathering labeled examples often constrain the amount of data that can be used for training. In settings where the number of training examples is limited, it often becomes meaningful to carefully see just which examples are Q1 selected. In active learning (AL), the model itself places a hands-on role in the selection of examples for labeling from a large pool of unlabeled examples. These examples are used for model training. Numerous studies have demonstrated, both empirically and theoretically, the benefits of AL: Given a fixed budget, a training system that interactively involves the current model in selecting the training examples can often result in a far greater accuracy than a system that simply selects random training examples. Imbalanced settings provide special opportunities and challenges for AL. For example, while AL can be used to build models that counteract the harmful effects of learning under class imbalance, extreme class imbalance can cause an AL strategy to "fail," preventing the selection scheme from choosing any useful examples for labeling. This chapter focuses on the interaction between AL and class imbalance, discussing (i) AL techniques designed specifically for dealing with imbalanced settings, (ii) strategies that leverage AL to overcome the deleterious effects of class imbalance, (iii) how extreme class imbalance can prevent AL systems from selecting useful examples, and alternatives to AL in these cases.

show abstract

“…Our work is motivated by Nguyen and Smeulders [12] and Donmez et al [13]. Philosophically, our work is most aligned with Sheng et al [14], where relabeling is used to obviate the effects of noise and Vijayanarasimhan et al [15], which identifies promising crowdsourcing annotation tasks given a specific data labeling budget.…”

Section: Related Workmentioning

confidence: 99%

Incremental Relabeling for Active Learning with Noisy Crowdsourced Annotations

Zhao

Sukthankar

2011

2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing

View full text Add to dashboard Cite

Abstract-Crowdsourcing has become an popular approach for annotating the large quantities of data required to train machine learning algorithms. However, obtaining labels in this manner poses two important challenges. First, naively labeling all of the data can be prohibitively expensive. Second, a significant fraction of the annotations can be incorrect due to carelessness or limited domain expertise of crowdsourced workers. Active learning provides a natural formulation to address the former issue by affordably selecting an appropriate subset of instances to label. Unfortunately, most active learning strategies are myopic and sensitive to label noise, which leads to poorly trained classifiers. We propose an active learning method that is specifically designed to be robust to such noise. We present an application of our technique in the domain of activity recognition for eldercare and validate the proposed approach using both simulated and realworld experiments using Amazon Mechanical Turk.

show abstract

Dual Strategy Active Learning

Cited by 131 publications

References 13 publications

Information Extraction with Active Learning: A Case Study in Legal Text

Information Extraction with Active Learning: A Case Study in Legal Text

Class Imbalance and Active Learning

Incremental Relabeling for Active Learning with Noisy Crowdsourced Annotations

Contact Info

Product

Resources

About