Learning to Few-Shot Learn Across Diverse Natural Language Classification Tasks

Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little

“…The present analyses, when taken in conjunction with our main results in §4.1, suggest that fine-tuning on large training datasets with complex classifiers in the pursuit of state-of-the-art results has mostly nullified the impact of word order in the pre-trained representations. Few shot (Bansal et al, 2019) and few sample (Zhang et al, 2021) learning and evaluation could potentially require more word order signal, and thereby encouraging the model to leverage its own learned syntax better.…”

Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning

“…We also include the SOTA results from [Bansal et al, 2020] for comparison and note that PACMAML is consistently the best performer over all three few-shot settings k = 4, 8, 16. In comparison, MAML and BMAML perform worse, possibly due to sensitivity to learning rates, as suggested by [Bansal et al, 2019]. Beyond generalization errors, in Table 2 (bottom) we also compare the memory usage of MAML/BMAML against PACMAML over different adaptive layer thresholds v. These results emphasize the computational advantage of PACMAML by showing that as more layers are adapted (lower v), MAML consumes significantly more memory due to its high-order derivatives.…”

“…Natural Language Inference Lastly, we evaluate the meta-learning algorithms on the large-scale BERT-base [Devlin et al, 2019] model containing 110M parameters. Our experiment involves 12 practical natural language inference tasks from [Bansal et al, 2019] Following [Bansal et al, 2019], we used the pretrained BERT-base model as our base model (hyperprior), and used GLUE benchmark tasks [Wang et al, 2018] for meta-training the models and meta-validation for hyperparameter search, before fine-tuning them for the 12 target tasks. The finetuning data contains k ∈ {4, 8, 16}-shot data for each class in each task.…”

“…For every k, 10 fine-tuning datasets were sampled for each target task. The final reported result is the average of the 10 models fine-tuned over these 10 datasets (for each task and each k separately), and evaluated on the entire test set for each target task [Bansal et al, 2019]. The data size of the observed tasks are fixed to be m i = 256, where the data points for each observed task are randomly sampled from the training data of one of the GLUE tasks.…”

“…Because the number of classes in these 12 few-shot tasks varies from 2 to 12, we choose the inner data size m i from {32, 64} for MAML, BMAML and PACMAML. As in [Bansal et al, 2019], we also partition the set of model parameters to task-specific and task-agnostic. For the 12-layer BERT-base model, we consider a hyper-parameter v ∈ {6, 9, 11, 12}, where only the layers higher than the v-th layer are considered task-specific and will be adapted in the inner loop.…”