Small-Text: Active Learning for Text Classification in Python

Abstract

We introduce small-text, an easy-to-use active learning library, which offers pool-based active learning for single-and multi-label text classification in Python. It features numerous pre-implemented state-of-the-art query strategies, including some that leverage the GPU. Standardized interfaces allow the combination of a variety of classifiers, query strategies, and stopping criteria, facilitating a quick mix and match, and enabling a rapid and convenient development of both active learning experiments and applications. With the objective of making various classifiers and query strategies accessible for active learning, small-text integrates several well-known machine learning libraries, namely scikit-learn, PyTorch, and Hugging Face transformers. The latter integrations are optionally installable extensions, so GPUs can be used but are not required. Using this new library, we investigate the performance of the recently published SetFit training paradigm, which we compare to vanilla transformer fine-tuning, finding that it matches the latter in classification accuracy while outperforming it in area under the curve. The library is available under the MIT License at https://github.com/webis-de/small-text, in version 1.3.0 at the time of writing.

A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing

“…Active Learning (AL), which contains several stages by definition, has also been shown to boost performance of BERT models for minority classes (Ein-Dor et al, 2020). For a discussion about AL and deep learning, see Schröder and Niekler (2020).…”

Classifying Speech Acts in Political Communication: A Transformer-based Approach with Weak Supervision and Active Learning

“…The goal is to intelligently choose data points for labelling, in order to improve the performance and efficiency of the learning process. Over the years, many query strategies have been proposed by various researchers, most of them using prediction based query strategies [23]. Active learning follows a process that involves two pools: a labelled pool and an unlabelled pool.…”