Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks

Abstract

Self-supervised pre-training of transformer models has revolutionized NLP applications. Such pre-training with language modeling objectives provides a useful initial point for parameters that generalize well to new tasks with fine-tuning. However, fine-tuning is still data inefficient -when there are few labeled examples, accuracy can be low. Data efficiency can be improved by optimizing pre-training directly for future fine-tuning with few examples; this can be treated as a meta-learning problem. However, standard meta-learning techniques require many training tasks in order to generalize; unfortunately, finding a diverse set of such supervised tasks is usually difficult. This paper proposes a self-supervised approach to generate a large, rich, metalearning task distribution from unlabeled text. This is achieved using a cloze-style objective, but creating separate multi-class classification tasks by gathering tokens-to-be blanked from among only a handful of vocabulary terms. This yields as many unique meta-training tasks as the number of subsets of vocabulary terms. We meta-train a transformer model on this distribution of tasks using a recent meta-learning framework. On 17 NLP tasks, we show that this meta-training leads to better few-shot generalization than language-model pre-training followed by finetuning. Furthermore, we show how the self-supervised tasks can be combined with supervised tasks for meta-learning, providing substantial accuracy gains over previous supervised meta-learning.

Learning to Learn to Disambiguate: Meta-Learning for Few-Shot Word Sense Disambiguation

“…Informed output layer initialization in Proto(FO)MAML is therefore important for effective learning in such scenarios. A similar problem with FOMAML is also pointed out by Bansal et al (2019), who design a differentiable parameter generator for the output layer.…”

“…Dou et al (2019) perform metatraining on certain high-resource tasks from the GLUE benchmark and metatest on certain low-resource tasks from the same benchmark. Bansal et al (2019) propose a softmax parameter generator component that can enable a varying number of classes in the meta-training tasks. They choose the tasks in GLUE along with SNLI (Bowman et al, 2015) for meta-training, and use entity typing, relation classification, sentiment classification, text categorization, and scientific NLI as the test tasks.…”

“…It has achieved success in computer vision (Triantafillou et al, 2020;Fontanini et al, 2020;Hendryx et al, 2019;Wang et al, 2020) and reinforcement learning (Wang et al, 2016;Duan et al, 2016;Alet et al, 2020). It has also recently made its way into NLP, and has been applied to machine translation (Gu et al, 2018), relation (Obamuyide and Vlachos, 2019b) and text (Yu et al, 2018) classification, and sentence-level semantic tasks (Dou et al, 2019;Bansal et al, 2019).…”

Task-Aware Representation of Sentences for Generic Text Classification

“…The problem of zero shot, and few shot learning has lately been proposed in the context of NLP Geng et al, 2019) using meta-learning. Model agnostic meta-learning (MAML) has been explored to tackle tasks with disjoint label spaces (Bansal et al, 2019). However, these models are not capable of making zero shot predictions.…”

“…Few-shot transfer learning. Real world text classification scenarios are often characterized by a lack of annotated corpora and rapidly changing information needs (Chiticariu et al, 2013), motivating research into methods that allow us to train text classifiers for new classes with only a handful of training examples (Bansal et al, 2019;Yogatama et al, 2019). In such cases, a standard approach is to transfer knowledge from an existing model for classification task X to initialize the weights for a model for the new classification task Y.…”