Practical, Efficient, and Customizable Active Learning for Named Entity Recognition in the Digital Humanities

Abstract

Scholars in inter-disciplinary fields like the Digital Humanities are increasingly interested in semantic annotation of specialized corpora. Yet, under-resourced languages, imperfect or noisily structured data, and user-specific classification tasks make it difficult to meet their needs using off-the-shelf models. Manual annotation of large corpora from scratch, meanwhile, can be prohibitively expensive. Thus, we propose an active learning solution for named entity recognition, attempting to maximize a custom model's improvement per additional unit of manual annotation. Our system robustly handles any domain or user-defined label set and requires no external resources, enabling quality named entity recognition for Humanities corpora where such resources are not available. Evaluating on typologically disparate languages and datasets, we reduce required annotation by 20-60% and greatly outperform a competitive active learning baseline.

Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates

“…Although the proposed solutions are shown to outperform other heuristic algorithms with comparably weak models (basic CRF or BERT without fine-tuning) in experiments with a small number of AL iterations, they can be not very practical due to the high computational costs of collecting training data for policy models. Other notable works on deep active learning include (Erdmann et al, 2019), which proposes an AL algorithm based on a bootstrapping approach (Jones et al, 1999) and (Lowell et al, 2019), which concerns the problem of the mismatch between a model used to construct a training dataset via AL (acquisition model) and a final model that is trained on it (successor model).…”

From Zero to Hero: Human-In-The-Loop Entity Linking in Low Resource Domains

“…We do not evaluate on a holdout set. Instead, we follow Erdmann et al (2019) and simulate annotating the complete corpus and evaluate on the very same data as we are interested in how an annotated sub-set helps to annotate the rest of the data, not how well the model generalizes. We assume that users annotate mention spans perfectly, i.e.…”

Named Entity Recognition and Classification on Historical Documents: A Survey

“…Another option is active learning, where a ML system asks an oracle (or a user) to select the most relevant examples to consider, thereby lowering the number of data points required to learn a model. This is the approach adopted by Erdmann et al [60] to recognise entities in various Latin classical texts, based on an active learning pipeline able to predict how many and which sentences need to be annotated to achieve a certain degree of accuracy, and later on released as toolkit to build custom NER models for the humanities [61]. Finally, another strategy is data augmentation, where an existing data set is expanded via the transformation of training instances without changing their label.…”