Continuous Active Learning Using Pretrained Transformers

Sadri, Nima; Cormack, Gordon V.

doi:10.48550/arxiv.2208.06955

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Text classifiers based on BioBERT (Lee et al, 2020) and PubMedBERT (Gu et al, 2020) were also developed. However, neither outperformed the linear logistic regression model on CLEF datasets, a result consistent with previous findings for TAR models (Yang et al, 2022;Sadri and Cormack, 2022).…”

Section: Methodssupporting

confidence: 90%

Combining Counting Processes and Classification Improves a Stopping Rule for Technology Assisted Review

Hezam,

Stevenson

2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Technology Assisted Review (TAR) stopping rules aim to reduce the cost of manually assessing documents for relevance by minimising the number of documents that need to be examined to ensure a desired level of recall. This paper extends an effective stopping rule using information derived from a text classifier that can be trained without the need for any additional annotation. Experiments on multiple data sets (CLEF e-Health, TREC Total Recall, TREC Legal and RCV1) showed that the proposed approach consistently improves performance and outperforms several alternative methods. 1

show abstract

Section: Methodssupporting

confidence: 90%

Combining Counting Processes and Classification Improves a Stopping Rule for Technology Assisted Review

Hezam,

Stevenson

2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

show abstract

“…A variation of the AL setting that has shown success in certain domain-specific tasks is that of continuous active learning [27,47,64], where documents are iteratively retrieved by actively learning for one specific query, typically aiming for total recall [42]. For the task of technology assisted review (TAR), Yang et al [63] propose a TAR cost framework, however this framework focuses on cost modeling for reviewing one specific query.…”

Section: Related Workmentioning

confidence: 99%

Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selection

Althammer,

Zuccon,

Hofstätter

et al. 2023

Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia P

View full text Add to dashboard Cite

Search methods based on Pretrained Language Models (PLM) have demonstrated great effectiveness gains compared to statistical and early neural ranking models. However, fine-tuning PLM-based rankers requires a great amount of annotated training data. Annotating data involves a large manual effort and thus is expensive, especially in domain specific tasks. In this paper we investigate finetuning PLM-based rankers under limited training data and budget. We investigate two scenarios: fine-tuning a ranker from scratch, and domain adaptation starting with a ranker already fine-tuned on general data, and continuing fine-tuning on a target dataset.We observe a great variability in effectiveness when fine-tuning on different randomly selected subsets of training data. This suggests that it is possible to achieve effectiveness gains by actively selecting a subset of the training data that has the most positive effect on the rankers. This way, it would be possible to fine-tune effective PLM rankers at a reduced annotation budget. To investigate this, we adapt existing Active Learning (AL) strategies to the task of fine-tuning PLM rankers and investigate their effectiveness, also considering annotation and computational costs. Our extensive analysis shows that AL strategies do not significantly outperform random selection of training subsets in terms of effectiveness. We further find that gains provided by AL strategies come at the expense of more assessments (thus higher annotation costs) and AL strategies underperform random selection when comparing effectiveness given a fixed annotation cost. Our results highlight that "optimal" subsets of training data that provide high effectiveness at low annotation cost do exist, but current mainstream AL strategies applied to PLM rankers are not capable of identifying them.

show abstract

Continuous Active Learning Using Pretrained Transformers

Cited by 2 publications

References 0 publications

Combining Counting Processes and Classification Improves a Stopping Rule for Technology Assisted Review

Combining Counting Processes and Classification Improves a Stopping Rule for Technology Assisted Review

Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selection

Contact Info

Product

Resources

About