WeakAL: Combining Active Learning and Weak Supervision

Gonsior, Julius; Thiele, Maik; Lehner, Wolfgang

doi:10.1007/978-3-030-61527-7_3

Cited by 6 publications

(2 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Weak supervision techniques obtain these noisy labels by tapping into heuristics (Ratner et al, 2017;Meng et al, 2018;Awasthi et al, 2020), feature annotation (Mann and McCallum, 2010), external knowledge bases (Hoffmann et al, 2011;Min et al, 2013), pretrained models (Bach et al, 2019;Zhang et al, 2021) and third-party tools (Lison et al, 2020). Moreover, weak supervision can be combined with the active learning framework (Gonsior et al, 2020) to select the most informative data to be annotated by humans and utilize weak supervision to decide noisy labels. Given LLMs' stunning zero-shot capabilities, our work explores the possibility of using them as a more efficient labeling source, thus freeing up resources to be reinvested in the research pipeline.…”

Section: Weak Supervisionmentioning

confidence: 99%

CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation

Li,

Shi,

Ziems

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Annotated data plays a critical role in Natural Language Processing (NLP) in training models and evaluating their performance. Given recent developments in Large Language Models (LLMs), models such as ChatGPT demonstrate zero-shot capability on many text-annotation tasks, comparable with or even exceeding human annotators. Such LLMs can serve as alternatives for manual annotation, due to lower costs and higher scalability. However, limited work has leveraged LLMs as complementary annotators, nor explored how annotation work is best allocated among humans and LLMs to achieve both quality and cost objectives. We propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of unstructured texts at scale. Under this framework, we utilize uncertainty to estimate LLMs' annotation capability. Our empirical study shows CoAnnotating to be an effective means to allocate work from results on different datasets, with up to 21% performance improvement over random baseline. For code implementation, see https: //github.com/SALT-NLP/CoAnnotating.

show abstract

Section: Weak Supervisionmentioning

confidence: 99%

CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation

Li,

Shi,

Ziems

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…Thanks to their simplicity, uncertainty-based methods belong to the most popular ones. Uncertainty-based methods can use least confidence scores [8,20,26], max margin scores [27,28], or max entropy scores [29] for querying.…”

Section: Related Work a Aspect-based Sentiment Analysismentioning

confidence: 99%

LETS: A Label-Efficient Training Scheme for Aspect-Based Sentiment Analysis by Using a Pre-Trained Language Model

Shim

Lowet²,

Luca

et al. 2021

IEEE Access

View full text Add to dashboard Cite

Recently proposed pre-trained language models can be easily fine-tuned to a wide range of downstream tasks. However, a large-scale labelled task-specific dataset is required for fine-tuning creating a bottleneck in the development process of machine learning applications. To foster a fast development by reducing manual labelling efforts, we propose a Label-Efficient Training Scheme (LETS). The proposed LETS consists of three elements: (i) task-specific pre-training to exploit unlabelled task-specific corpus data, (ii) label augmentation to maximise the utility of labelled data, and (iii) active learning to label data strategically. In this paper, we apply LETS to a novel aspect-based sentiment analysis (ABSA) use-case for analysing the reviews of the health-related program supporting people to improve their sleep quality. We validate the proposed LETS on a custom health-related program-reviews dataset and another ABSA benchmark dataset. Experimental results show that the LETS can reduce manual labelling efforts 2-3 times compared to labelling with random sampling on both datasets. The LETS also outperforms other state-of-the-art active learning methods. Furthermore, the experimental results show that LETS can contribute to better generalisability with both datasets compared to other methods thanks to the task-specific pre-training and the proposed label augmentation. We expect this work could contribute to the natural language processing (NLP) domain by addressing the issue of the high cost of manually labelling data. Also, our work could contribute to the healthcare domain by introducing a new potential application of NLP techniques.

show abstract

Scoping Review of Active Learning Strategies and Their Evaluation Environments for Entity Recognition Tasks

Kohl,

Krämer,

Fohry

et al. 2024

Communications in Computer and Information Science

View full text Add to dashboard Cite

WeakAL: Combining Active Learning and Weak Supervision

Cited by 6 publications

References 10 publications

CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation

CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation

LETS: A Label-Efficient Training Scheme for Aspect-Based Sentiment Analysis by Using a Pre-Trained Language Model

Scoping Review of Active Learning Strategies and Their Evaluation Environments for Entity Recognition Tasks

Contact Info

Product

Resources

About