Semi-supervised Meta-learning for Cross-domain Few-shot Intent Classification

Abstract

Meta-learning aims to optimize the model's capability to generalize to new tasks and domains. Lacking a data-efficient way to create meta training tasks has prevented the application of meta-learning to the real-world few shot learning scenarios. Recent studies have proposed unsupervised approaches to create meta-training tasks from unlabeled data for free, e.g., the SMLMT method (Bansal et al., 2020a) constructs unsupervised multiclass classification tasks from the unlabeled text by randomly masking words in the sentence and let the meta learner choose which word to fill in the blank. This study proposes a semi-supervised meta-learning approach that incorporates both the representation power of large pre-trained language models and the generalization capability of prototypical networks enhanced by SMLMT. The semi-supervised meta training approach avoids overfitting prototypical networks on a small number of labeled training examples and quickly learns cross-domain task-specific representation only from a few supporting examples. By incorporating SMLMT with prototypical networks, the meta learner generalizes better to unseen domains and gains higher accuracy on out-ofscope examples without the heavy lifting of pre-training. We observe significant improvement in few-shot generalization after training only a few epochs on the intent classification tasks evaluated in a multi-domain setting.

Advances and Challenges in Meta-Learning: A Technical Review

“…In [102], the authors propose a transferable meta-learning algorithm with a meta task adaptation to minimize the domain divergence and thus facilitate knowledge transfer across domains. To further improve the transferability of cross-domain knowledge, [103] and [104] propose to incorporate semi-supervised techniques into the meta-learning framework. Specifically, [103] combines the representation power of large pre-trained language models (e.g., BERT [33]) with the generalization capability of prototypical networks enhanced by SMLMT [105] to achieve effective generalization and adaptation to tasks from new domains.…”

Learning Adaptive Probabilistic Models for Uncertainty-Aware Air Pollution Prediction

“…Therefore our model not only provides the adaptive ability to the given backbone predictor as other meta-learning works but simultaneously provides better probabilistic modeling for uncertainty-aware applications. There are also previous works that combine adversarial training and meta learning topic [38], [39], [40], [41]. Our work is different because we adversarially train the generative model itself as a meta learner instead of as data-augmentation tricks, as pretrained models, or for adversarial robustness.…”

Task‐oriented feature hallucination for few‐shot image classification

“…ICI [33] and the method of Lazarou et al. [34] progressively filter out less trustworthy instances to perform pseudo‐labelling, LST [35] weighted the cherry‐picked pseudo‐labelled data further.…”

“…Knowledge distillation has been employed in FSL methods [9, 13, 18, 34, 35, 41, 42]. These methods typically adopt the model compression strategy, that is, under the guidance of a teacher model to build the student model.…”

“…These methods typically adopt the model compression strategy, that is, under the guidance of a teacher model to build the student model. Techniques such as pseudo‐labels are used to guide the knowledge distillation process for more general embeddings [13, 34, 35], or hallucinator [9, 42] proposes a learning paradigm where the knowledge is distilled in a self‐supervised fashion. SKD [41] mines auxiliary self‐supervised learning (SSL) signals from the limited data to learn the output‐space manifold of each class.…”