Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks

Bansal, Trapit; Jha, Rishikesh; Munkhdalai, Tsendsuren; McCallum, Andrew

doi:10.48550/arxiv.2009.08445

Cited by 5 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The detailed results of the 12 NLI tasks, their standard errors, as well as the hyperparameter selections are all included in the Appendix. We also include the SOTA results from [Bansal et al, 2020] for comparison and note that PACMAML is consistently the best performer over all three few-shot settings k = 4, 8, 16. In comparison, MAML and BMAML perform worse, possibly due to sensitivity to learning rates, as suggested by [Bansal et al, 2019].…”

Section: Few-shot Classification Problemsmentioning

confidence: 99%

Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning

Ding¹,

Chen²,

Levinboim³

et al. 2021

Preprint

View full text Add to dashboard Cite

Despite recent advances in its theoretical understanding, there still remains a significant gap in the ability of existing PAC-Bayesian theories on meta-learning to explain performance improvements in the few-shot learning setting, where the number of training examples in the target tasks is severely limited. This gap originates from an assumption in the existing theories which supposes that the number of training examples in the observed tasks and the number of training examples in the target tasks follow the same distribution, an assumption that rarely holds in practice. By relaxing this assumption, we develop two PAC-Bayesian bounds tailored for the few-shot learning setting and show that two existing meta-learning algorithms (MAML and Reptile) can be derived from our bounds, thereby bridging the gap between practice and PAC-Bayesian theories. Furthermore, we derive a new computationally-efficient PACMAML algorithm, and show it outperforms existing meta-learning algorithms on several few-shot benchmark datasets.

show abstract

Section: Few-shot Classification Problemsmentioning

confidence: 99%

Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning

Ding¹,

Chen²,

Levinboim³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…BERT or RoBERTa). However, we did not adopt these approaches for the following reasons: (1) We observed that the perturbed examples from such adversarial methods are often unnatural and not readable by humans. (2) Both adversarial perturbation and selection require a reference model, which violates our model-agnostic task formulation principle in Table 2.…”

Section: Sampling Of Training and Test Datamentioning

confidence: 99%

“…Another line of work explored semi-supervised learning; where unlabeled data, alongside usually small amounts of labeled data, is used for learning [35,37,15,17]. Recent studies have also explored meta-learning in NLU where the models have access to data from many training tasks to learn from, and evaluate the few-shot learning ability on unseen test tasks [4,1,20]. In this work, we do not address the meta-learning setting [40].…”

Section: Introductionmentioning

confidence: 99%

CLUES: Few-Shot Learning Evaluation in Natural Language Understanding

Mukherjee¹,

Liu²,

Zheng³

et al. 2021

Preprint

View full text Add to dashboard Cite

Most recent progress in natural language understanding (NLU) has been driven, in part, by benchmarks such as GLUE, SuperGLUE, SQuAD, etc. In fact, many NLU models have now matched or exceeded "human-level" performance on many tasks in these benchmarks. Most of these benchmarks, however, give models access to relatively large amounts of labeled data for training. As such, the models are provided far more data than required by humans to achieve strong performance. That has motivated a line of work that focuses on improving few-shot learning performance of NLU models. However, there is a lack of standardized evaluation benchmarks for few-shot NLU resulting in different experimental settings in different papers. To help accelerate this line of work, we introduce CLUES 1 , a benchmark for evaluating the few-shot learning capabilities of NLU models. We demonstrate that while recent models reach human performance when they have access to large amounts of labeled data, there is a huge gap in performance in the few-shot setting for most tasks. We also demonstrate differences between alternative model families and adaptation techniques in the few shot setting. Finally, we discuss several principles and choices in designing the experimental settings for evaluating the true few-shot learning performance and suggest a unified standardized approach to few-shot learning evaluation. We aim to encourage research on NLU models that can generalize to new tasks with a small number of examples. Code and data for CLUES are available at https://github.com/microsoft/CLUES.We introduce a new few-shot NLU benchmark (CLUES), that aims to address these limitations. Few-shot evaluation of NLU performance has emerged as an important task and is considered to 1 Constrained Language Understanding Evaluation Standard 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks.

show abstract

“…Prototype-based methods recently become popular few-shot learning approaches in machine learning community. It was firstly studied in the context of image classification (Vinyals et al, 2016;Sung et al, 2018;Zhao et al, 2020), and has recently been adapted to different NLP tasks such as text classification Geng et al, 2019;Bansal et al, 2020), machine translation (Gu et al, 2018) and relation classification (Han et al, 2018).…”

Section: Self-trainingmentioning

confidence: 99%

Few-Shot Named Entity Recognition: A Comprehensive Study

Huang¹,

Li²,

Subudhi³

et al. 2020

Preprint

View full text Add to dashboard Cite

This paper presents a comprehensive study to efficiently build named entity recognition (NER) systems when a small number of indomain labeled data is available. Based upon recent Transformer-based self-supervised pretrained language models (PLMs), we investigate three orthogonal schemes to improve the model generalization ability for few-shot settings: (1) meta-learning to construct prototypes for different entity types, (2) supervised pre-training on noisy web data to extract entity-related generic representations and (3) self-training to leverage unlabeled in-domain data. Different combinations of these schemes are also considered. We perform extensive empirical comparisons on 10 public NER datasets with various proportions of labeled data, suggesting useful insights for future research. Our experiments show that (i) in the few-shot learning setting, the proposed NER schemes significantly improve or outperform the commonly used baseline, a PLM-based linear classifier fine-tuned on domain labels. (ii) We create new state-of-the-art results on both few-shot and training-free settings compared with existing methods. We will release our code and pretrained models for reproducible research.

show abstract

Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks

Cited by 5 publications

References 0 publications

Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning

Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning

CLUES: Few-Shot Learning Evaluation in Natural Language Understanding

Few-Shot Named Entity Recognition: A Comprehensive Study

Contact Info

Product

Resources

About