Active Learning with Subsequence Sampling Strategy for Sequence Labeling Tasks

Abstract

We propose an active learning framework for sequence labeling tasks. In each iteration, a set of subsequences are selected and manually labeled, while the other parts of sequences are left unannotated. The learning will stop automatically when the training data between consecutive iterations does not significantly change. We evaluate the proposed framework on chunking and named entity recognition data provided by CoNLL. Experimental results show that we succeed in obtaining the supervised F 1 only with 6.98%, and 7.01% of tokens being annotated, respectively.

A Little Annotation does a Lot of Good: A Study in Bootstrapping Low-resource Named Entity Recognizers

“…This way, the annotators only need to assign types to the chosen subspans without having to read and annotate the full sequence. To cope with the resulting partial annotation of sequences, we apply a constrained version of conditional random fields (CRFs), partial CRFs, during training that only learn from the annotated subspans (Tsuboi et al, 2008;Wanvarie et al, 2011). To evaluate our proposed methods, we conducted simulated active learning experiments on 5 languages: Spanish, Dutch, German, Hindi and Indonesian.…”

An Experimental Comparison of Active Learning Strategies for Partially Labeled Sequences

“…The main AL works in this latter line of work are (Shen et al, 2004), ) and (Wanvarie et al, 2011). Shen et al (2004) adopted SVMs as learning algorithm and proposed two strategies that combine three criteria, informativeness, representativeness and diversity.…”

“…At this point an AL strategy S will select a number of examples B that once labeled will hopefully improve the performance of the next classifier Φ i+1 . Algorithm 1 shows the pool-based AL framework for partially annotated sequences as introduced in (Wanvarie et al, 2011). Differently from AL for fully labeled sequences , thanks to the finer granularity of the partially labeled model, we use the token as basic annotation unit, instead of the entire sequence.…”

“…The Minimum Viterbi Probability (MVP) is the base strategy adopted in (Wanvarie et al, 2011). It takes as measure of informativeness the probability of the label chosen by the Viterbi algorithm.…”

“…A few works have investigated this idea. For instance, Wanvarie et al (2011) proposed to use Partially-Labeled Conditional Random Fields (PL-CRFs) (Tsuboi et al, 2008), a semi-supervised variation of Conditional Random Fields (CRFs) (Lafferty et al, 2001) able to deal with partially-labeled sequences, thus enabling to adopt as annotation unit single tokens and still learning from full sequences. AL with partially labeled sequences has proven to be effective in substantially reducing the amount of annotated data with respect to common AL approaches (see (Wanvarie et al, 2011)).…”