MetaPix: Domain transfer for semantic segmentation by meta pixel weighting

Jian, Yiren; Gao, Chongyang

doi:10.1016/j.imavis.2021.104334

“…Few-shot Learning is often tackled by meta learning (Li and Zhang, 2021;Bansal et al, 2020;Sharaf et al, 2020;Jian et al, 2020;Jian and Gao, 2021), data augmentation Jian and Torresani, 2022;Arthaud et al, 2021;Wei et al, 2021; Encoder MLM head SupCon(+) SupCon(-)…”

Section: Related Work and Backgroundmentioning

confidence: 99%

Contrastive Learning for Prompt-based Few-shot Language Learners

Jian¹,

Gao²,

Vosoughi³

2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

The impressive performance of GPT-3 using natural language prompts and in-context learning has inspired work on better fine-tuning of moderately-sized models under this paradigm. Following this line of work, we present a contrastive learning framework that clusters inputs from the same class for better generality of models trained with only limited examples. Specifically, we propose a supervised contrastive framework that clusters inputs from the same class under different augmented "views" and repel the ones from different classes. We create different "views" of an example by appending it with different language prompts and contextual demonstrations. Combining a contrastive loss with the standard masked language modeling (MLM) loss in prompt-based few-shot learners, the experimental results show that our method can improve over the state-of-the-art methods in a diverse set of 15 language tasks. Our framework makes minimal assumptions on the task or the base model, and can be applied to many recent methods with little modification. The code will be made available at: https://github.com/yiren-jian/LM-SupCon.

show abstract

“…Few-shot Learning is often tackled by meta learning (Li and Zhang, 2021;Bansal et al, 2020;Sharaf et al, 2020;Jian et al, 2020;Jian and Gao, 2021), data augmentation Jian and Torresani, 2022;Arthaud et al, 2021;Wei et al, 2021; Encoder MLM head SupCon(+) SupCon(-)…”

Section: Related Work and Backgroundmentioning

confidence: 99%

Contrastive Learning for Prompt-Based Few-Shot Language Learners

Jian¹,

Gao²,

Vosoughi³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

The impressive performance of GPT-3 using natural language prompts and in-context learning has inspired work on better fine-tuning of moderately-sized models under this paradigm. Following this line of work, we present a contrastive learning framework that clusters inputs from the same class for better generality of models trained with only limited examples. Specifically, we propose a supervised contrastive framework that clusters inputs from the same class under different augmented "views" and repel the ones from different classes. We create different "views" of an example by appending it with different language prompts and contextual demonstrations. Combining a contrastive loss with the standard masked language modeling (MLM) loss in prompt-based few-shot learners, the experimental results show that our method can improve over the state-of-the-art methods in a diverse set of 15 language tasks. Our framework makes minimal assumptions on the task or the base model, and can be applied to many recent methods with little modification. The code will be made available at: https://github.com/yiren-jian/LM-SupCon.

show abstract

“…Label Hallucination (Jian and Torresani, 2022) assigns soft pseudo-labels for unlabelled images to extend the fine-tuning few-shot dataset. Learning from limited labeled data (few-shot learning) in Computer Vision is usually achieved by meta-learning (Ren et al, 2018a,b;Jian et al, 2020;Jian and Gao, 2021) or transfer learning (Tian et al, 2020). In NLP, few-shot learning has been successfully applied to machine translation (Arthaud et al, 2021), abstract summarizing (Fabbri et al, 2021), question and answering (Hua et al, 2020;Ram et al, 2021), and entity recognition (de Lichy et al, 2021;Tong et al, 2021;Ding et al, 2021), by meta learning (Li and Zhang, 2021;Bansal et al, 2020;Sharaf et al, 2020), data augmentation (Wei et al, 2021;Wei and Zou, 2019;Karimi et al, 2021;, and prompts Tam et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

Embedding Hallucination for Few-Shot Language Fine-tuning

Jian¹,

Gao²,

Vosoughi³

2022

Preprint

Self Cite

0

View full text Add to dashboard Cite

Few-shot language learners adapt knowledge from a pre-trained model to recognize novel classes from a few-labeled sentences. In such settings, fine-tuning a pre-trained language model can cause severe over-fitting. In this paper, we propose an Embedding Hallucination (EmbedHalluc) method, which generates auxiliary embedding-label pairs to expand the finetuning dataset. The hallucinator is trained by playing an adversarial game with the discriminator, such that the hallucinated embedding is indiscriminative to the real ones in the finetuning dataset. By training with the extended dataset, the language learner effectively learns from the diverse hallucinated embeddings to overcome the over-fitting issue. Experiments demonstrate that our proposed method is effective in a wide range of language tasks, outperforming current fine-tuning methods. Further, we show that EmbedHalluc outperforms other methods that address this over-fitting problem, such as common data augmentation, semi-supervised pseudo-labeling, and regularization. The code will be made available at: https://github.com/yiren-jian/EmbedHalluc.

show abstract

MetaPix: Domain transfer for semantic segmentation by meta pixel weighting

Cited by 5 publications

References 5 publications

Contrastive Learning for Prompt-based Few-shot Language Learners

Contrastive Learning for Prompt-based Few-shot Language Learners

Contrastive Learning for Prompt-Based Few-Shot Language Learners

Embedding Hallucination for Few-Shot Language Fine-tuning

Contact Info

Product

Resources

About