Diverse Distributions of Self-Supervised Tasks for Meta-Learning in NLP

Abstract

Meta-learning considers the problem of learning an efficient learning process that can leverage its past experience to accurately solve new tasks. However, the efficacy of meta-learning crucially depends on the distribution of tasks available for training, and this is often assumed to be known a priori or constructed from limited supervised datasets. In this work, we aim to provide task distributions for meta-learning by considering self-supervised tasks automatically proposed from unlabeled text, to enable large-scale meta-learning in NLP. We design multiple distributions of self-supervised tasks by considering important aspects of task diversity, difficulty, type, domain, and curriculum, and investigate how they affect meta-learning performance. Our analysis shows that all these factors meaningfully alter the task distribution, some inducing significant improvements in downstream few-shot accuracy of the metalearned models. Empirically, results on 20 downstream tasks show significant improvements in few-shot learning -adding up to +4.2% absolute accuracy (on average) to the previous unsupervised meta-learning method, and perform comparably to supervised methods on the FewRel 2.0 benchmark.

Less is More? Investigating Meta-Learning’s Suitability in Sentence Compression for Low-Resource Data

“…The objective of assuming this distribution is to sample T i tasks from the P(T ) distribution -where T i is a task composed of a training set (D T rain T i , also known as support set) and a test set (D T est T i , also known as query set) -to train a meta-model that can generalize well to all tasks used in the training process. The trained meta-model can then be fine-tuned to a task T ′ , also sampled from the P(T ) distribution, that was not seen in the meta-model training [Bansal et al 2021]. When dealing with supervised learning tasks using meta-learning, we create the P(T ) distribution based on a fixed set of tasks, subsampled from all classes [Vinyals et al 2016].…”

Meta Learning and Its Applications to Natural Language Processing

“…Inspired from the self-supervised learning, Bansal et al (2020b) generates a large number of cloze tasks, which can be considered as multi-class classification tasks but obtained without labeling effort, to augment the meta-training tasks. Bansal et al (2021) further explores the influence of unsupervised task distribution and creates task distributions that are inductive to better meta-training efficacy. The self-supervised generated tasks improve the performance on a wide range of different meta-testing tasks which are classification problems (Bansal et al, 2020b), and it even performs comparably with supervised meta-learning methods on FewRel 2.0 benchmark (Gao et al, 2019b) on 5-shot evaluation (Bansal et al, 2021).…”

“…Bansal et al (2021) further explores the influence of unsupervised task distribution and creates task distributions that are inductive to better meta-training efficacy. The self-supervised generated tasks improve the performance on a wide range of different meta-testing tasks which are classification problems (Bansal et al, 2020b), and it even performs comparably with supervised meta-learning methods on FewRel 2.0 benchmark (Gao et al, 2019b) on 5-shot evaluation (Bansal et al, 2021).…”