To Softmax, or not to Softmax: that is the question when applying Active Learning for Transformer Models

Gonsior, Julius; Falkenberg, Christian; Magino, Silvio; Reusch, Anja; Thiele, Maik; Lehner, Wolfgang

doi:10.48550/arxiv.2210.03005

Cited by 3 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The initial studies showed promise, with AL methods outperforming random sampling for text classification (Dor et al, 2020;Grießhaber et al, 2020). The field is gradually gaining traction with studies demonstrating AL effectiveness even with simple uncertainty-based methods (Gonsior et al, 2022;Schröder et al, 2022). Moreover, PLMs open up new possibilities, such as complementing AL with model adaptation using unlabeled data (Yuan et al, 2020;Margatina et al, 2022).…”

Section: Related Workmentioning

confidence: 99%

On Dataset Transferability in Active Learning for Transformers

Jelenić,

Jukić,

Drobac

et al. 2023

Findings of the Association for Computational Linguistics: ACL 2023

View full text Add to dashboard Cite

Active learning (AL) aims to reduce labeling costs by querying the examples most beneficial for model learning. While the effectiveness of AL for fine-tuning transformer-based pre-trained language models (PLMs) has been demonstrated, it is less clear to what extent the AL gains obtained with one model transfer to others. We consider the problem of transferability of actively acquired datasets in text classification and investigate whether AL gains persist when a dataset built using AL coupled with a specific PLM is used to train a different PLM. We link the AL dataset transferability to the similarity of instances queried by the different PLMs and show that AL methods with similar acquisition sequences produce highly transferable datasets regardless of the models used. Additionally, we show that the similarity of acquisition sequences is influenced more by the choice of the AL method than the choice of the model.

show abstract

Section: Related Workmentioning

confidence: 99%

On Dataset Transferability in Active Learning for Transformers

Jelenić,

Jukić,

Drobac

et al. 2023

Findings of the Association for Computational Linguistics: ACL 2023

View full text Add to dashboard Cite

show abstract

“…They evaluated this realworld dataset both as a single-and multi-label active learning setup, finding that active learning can considerably reduce the annotation efforts. Gonsior et al (2022) examined several alternatives to the softmax function to obtain better confidence estimates for active learning. Their setup extended small-text to incorporate additional softmax alternatives and found that confidence-based methods mostly selected outliers.…”

Section: Classification Of Citizens' Contributionsmentioning

confidence: 99%

Small-Text: Active Learning for Text Classification in Python

Schröder,

Müller,

Niekler

et al. 2023

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrati

View full text Add to dashboard Cite

We introduce small-text, an easy-to-use active learning library, which offers pool-based active learning for single-and multi-label text classification in Python. It features numerous pre-implemented state-of-the-art query strategies, including some that leverage the GPU. Standardized interfaces allow the combination of a variety of classifiers, query strategies, and stopping criteria, facilitating a quick mix and match, and enabling a rapid and convenient development of both active learning experiments and applications. With the objective of making various classifiers and query strategies accessible for active learning, small-text integrates several well-known machine learning libraries, namely scikit-learn, PyTorch, and Hugging Face transformers. The latter integrations are optionally installable extensions, so GPUs can be used but are not required. Using this new library, we investigate the performance of the recently published SetFit training paradigm, which we compare to vanilla transformer fine-tuning, finding that it matches the latter in classification accuracy while outperforming it in area under the curve. The library is available under the MIT License at https://github.com/webis-de/small-text, in version 1.3.0 at the time of writing.

show abstract