Exploring the Efficiency of Batch Active Learning for Human-in-the-Loop Relation Extraction

Abstract

Domain-specific relation extraction requires training data for supervised learning models, and thus, significant labeling effort. Distant supervision is often leveraged for creating large annotated corpora however these methods require handling the inherent noise. On the other hand, active learning approaches can reduce the annotation cost by selecting the most beneficial examples to label in order to learn a good model. The choice of examples can be performed sequentially, i.e. select one example in each iteration, or in batches, i.e. select a set of examples in each iteration. The optimization of the batch size is a practical problem faced in every real-world application of active learning, however it is often treated as a parameter decided in advance. In this work, we study the trade-off between model performance, the number of requested labels in a batch and the time spent in each round for real-time, domain specific relation extraction. Our results show that the use of an appropriate batch size produces competitive performance, even compared to a fully sequential strategy, while reducing the training time dramatically.

Using Active Learning Methods to Strategically Select Essays for Automated Scoring

“…Batch‐mode AL is a practical technique where the most informative essays are identified in each training iteration. Batch‐mode AL selection serves as an improvement over single instance selection because by sequentially selecting a single essay in each training iteration, a set of essays can be selected over all of the iterations (Lourentzou et al., 2018). The general workflow for batch‐model AL begins with a given set of training data that contain scores where a model is built and fit to the training data.…”

Task-Driven Privacy-Preserving Data-Sharing Framework for the Industrial Internet

“…Other works focus on interleaving processes to reduce annotator waiting time in batch active learning [38], and active learning domain adaptation settings by clustering uncertaintyweighted embeddings [39] or by utilizing reinforcement learning [40], Bayesian Optimization [41], and domain similarity metrics [42]. Recent works formulate active learning as a multi-armed bandit problem and select data from a set of candidates in each round [43,44,45].…”

Active Learning Strategy for COVID-19 Annotated Dataset

“…Global methods try to find the most informative set of samples from the whole space directly by solving an optimization problem [20], [24]- [28]. These approaches have mathematically and empirically demonstrated a good performance, however, they do not scale well with big datasets [29]. On the other hand, clusteringbased methods, which are highly scalable, partition either whole [30] or a fraction of (i.e.…”

“…The main objective of DS3 is to develop a scalable batchmodel framework for the class-imbalance problem. The success of batch mode active learning (BMAL) depends on selecting representative samples [37] as well as the batch size and total budget constraints [29]. The key question is how to find the most representative samples from both the minority and majority classes to cover the whole uncertain space given the limited budget.…”