Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates

Abstract

Annotating training data for sequence tagging of texts is usually very time-consuming. Recent advances in transfer learning for natural language processing in conjunction with active learning open the possibility to significantly reduce the necessary annotation budget. We are the first to thoroughly investigate this powerful combination for the sequence tagging task. We conduct an extensive empirical study of various Bayesian uncertainty estimation methods and Monte Carlo dropout options for deep pretrained models in the active learning framework and find the best combinations for different types of models. Besides, we also demonstrate that to acquire instances during active learning, a full-size Transformer can be substituted with a distilled version, which yields better computational performance and reduces obstacles for applying deep active learning in practice.

FAMIE: A Fast Active Learning Framework for Multilingual Information Extraction

“…We follow the AL settings in previous work to achieve consistent evaluation (Kim, 2020;Shelmanov et al, 2021;Liu et al, 2022). Specifically, the unlabeled pool is created by discarding labels from the original training data of each dataset; 2% of which (∼ 242 sentences) is selected for labeling at each iteration for a total of 25 iterations (examples of the first iteration are randomly sampled to serve as the seed D 0 ).…”

“…As discussed in the introduction, model training and data selection at each iteration of traditional AL methods might consume significant time (especially with the current trend of large-scale language models), thus introducing a long idle time for annotators that might reduce annotation quality and quantity. To this end, (Shelmanov et al, 2021) have explored approaches to accelerate training and data selection steps for AL by leveraging smaller and approximate models during the AL iterations. To make it more efficient, the main large model is only trained once in the end over all the annotated examples in AL.…”

“…However, an issue with these approaches is the inability to exploit recent advances in large-scale language models to achieve optimal performance. In addition, some recent works have also explored large-scale language models for AL (Shelmanov et al, 2021;Yuan et al, 2020); however, to reduce waiting time for annotators, such methods need to exclude the training of the large models in the AL iterations or employ small models for data selection, thus suffering from a harmful mismatch between the annotated examples and the main models (Lowell et al, 2019).…”

“…Unfortunately, despite much potentials, existing AL methods and frameworks are still not applied widely in practice due to their main focus on devising the most effective example selection algorithm for human annotation, e.g., based on the diversity of the examples (Shen et al, 2017;Yuan et al, 2020) and/or the uncertainty of the models (Roth and Small, 2006;Wang and Shang, 2014;Shelmanov et al, 2021). Training and selection time in the first and second steps of each AL interaction is thus not considered in prior work for sequence labeling.…”

Active Learning Helps Pretrained Models Learn the Intended Task

“…AL and example selection Active learning (AL) (Lewis and Catlett, 1994;Settles and Craven, 2008;Settles, 2009;Houlsby et al, 2011) is a well-studied field that investigates how machine learning algorithms might automatically select helpful additional data points to maximize their performance. Such strategies are especially helpful in imbalanced settings (Ertekin et al, 2007;Mussmann et al, 2020) and have been fruitfully applied to deep models (Gal et al, 2017;Beluch et al, 2018), including pretrained models (Yuan et al, 2020;Margatina et al, 2021;Shelmanov et al, 2021). Past work has also considered AL for few-shot learning (Woodward and Finn, 2017).…”