Want To Reduce Labeling Cost? GPT-3 Can Help

Wang, Shuohang; Liu, Yang; Yi‐chong, Xu; C, Zhu; Zeng, Michael

doi:10.18653/v1/2021.findings-emnlp.354

Cited by 65 publications

(46 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Potential impact on annotation campaigns. The performance obtained by the proposed approach highlight how in-context learning techniques may be used side-by-side with human experts within annotation campaigns [31].In general, the annotation of natural language texts describing business process models is a complicated task to be performed from scratch. A potential impact of approaches like the ones discussed in this paper is to support the annotation task by providing candidate annotations related to the process elements for which the performance are acceptable, e.g.…”

Section: Discussionmentioning

confidence: 98%

Leveraging pre-trained language models for conversational information seeking from text

Bellan¹,

Dragoni²,

Ghidini³

2022

Preprint

View full text Add to dashboard Cite

Recent advances in Natural Language Processing, and in particular on the construction of very large pre-trained language representation models, is opening up new perspectives on the construction of conversational information seeking (CIS) systems. In this paper we investigate the usage of in-context learning and pre-trained language representation models to address the problem of information extraction from process description documents, in an incremental question and answering oriented fashion. In particular we investigate the usage of the native GPT-3 (Generative Pre-trained Transformer 3) model, together with two in-context learning customizations that inject conceptual definitions and a limited number of samples in a few shot-learning fashion. The results highlight the potential of the approach and the usefulness of the in-context learning customizations, which can substantially contribute to address the "training data challenge" of deep learning based NLP techniques the BPM field. It also highlight the challenge posed by control flow relations for which further training needs to be devised.

show abstract

Section: Discussionmentioning

confidence: 98%

Leveraging pre-trained language models for conversational information seeking from text

Bellan¹,

Dragoni²,

Ghidini³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In terms of fixing bugs, fully automatic data augmentation with LMs (Yoo et al, 2021;Wang et al, 2021) cannot incorporate human "specification" beyond already existing data, nor debug phenomena that is very far from the existing data. On the other hand, general purpose or contrastive counterfactuals have shown mixed or marginally positive results (Huang et al, 2020; similar to what we observed in Section 3.2, except when large quantities of data are gathered (Nie et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

“…Out of these, only crowdsourcing can potentially fix bugs when enough data is gathered. On the other hand, fully automated approaches such as perturbations (Belinkov and Bisk, 2018;Prabhakaran et al, 2019), automatic adversarial examples (Ribeiro et al, 2018), and unguided data augmentation (Yoo et al, 2021;Wang et al, 2021) are severely restricted to specific kinds of problems (e.g. Ribeiro et al (2018) only deal with inconsistent predictions on paraphrases).…”

Section: Introductionmentioning

confidence: 99%

Adaptive Testing and Debugging of NLP Models

Ribeiro¹,

Lundberg²

2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

Current approaches to testing and debugging NLP models rely on highly variable human creativity and extensive labor, or only work for a very restrictive class of bugs. We present AdaTest, a process which uses large scale language models (LMs) in partnership with human feedback to automatically write unit tests highlighting bugs in a target model. Such bugs are then addressed through an iterative text-fixretest loop, inspired by traditional software development. In experiments with expert and non-expert users and commercial / research models for 8 different tasks, AdaTest makes users 5-10x more effective at finding bugs than current approaches, and helps users effectively fix bugs without adding new bugs. * Equal contribution, author order chosen by casting lots.

show abstract

“…Another line of work uses the outputs from a prompted language model as weak labels, as we do in this work. Wang et al (2021) propose to train smaller models on labels from GPT-3 to reduce annotation cost, but they train from individual, uncalibrated prompts and do not attempt to refine the prompt model alongside the smaller model. fine-tune a separate RoBERTa model for each prompt using a small amount of labeled data.…”

Section: Related Workmentioning

confidence: 99%

Co-training Improves Prompt-based Learning for Large Language Models

Lang¹,

Agrawal²,

Kim³

et al. 2022

Preprint

View full text Add to dashboard Cite

We demonstrate that co-training (Blum & Mitchell, 1998) can improve the performance of promptbased learning by using unlabeled data. While prompting has emerged as a promising paradigm for few-shot and zero-shot learning, it is often brittle and requires much larger models compared to the standard supervised setup. We find that cotraining makes it possible to improve the original prompt model and at the same time learn a smaller, downstream task-specific model. In the case where we only have partial access to a prompt model (e.g., output probabilities from GPT-3 (Brown et al., 2020)) we learn a calibration model over the prompt outputs. When we have full access to the prompt model's gradients but full finetuning remains prohibitively expensive (e.g., T0 (Sanh et al., 2022)), we learn a set of soft prompt continuous vectors to iteratively update the prompt model. We find that models trained in this manner can significantly improve performance on challenging datasets where there is currently a large gap between prompt-based learning and fully-supervised models.

show abstract

Want To Reduce Labeling Cost? GPT-3 Can Help

Cited by 65 publications

References 8 publications

Leveraging pre-trained language models for conversational information seeking from text

Leveraging pre-trained language models for conversational information seeking from text

Adaptive Testing and Debugging of NLP Models

Co-training Improves Prompt-based Learning for Large Language Models

Contact Info

Product

Resources

About