A Simple Weakly-Supervised Contrastive Learning Framework for Few-shot Sentiment Classification

Abstract

Most existing deep learning-based sentiment classification methods need large human-annotated data, but labeling large amounts of high-quality emotional texts is labor-intensive. Users on various social platforms generate massive amounts of tagged opinionated text (e.g., tweets, customer reviews), providing a new resource for training deep models. However, some of the tagged instances have sentiment tags that are diametrically opposed to their true semantics. We cannot use this tagged data directly because the noisy labeled instances have a negative impact on the training phase. In this paper, we present a novel Simple Weakly-supervised Contrastive Learning framework (SWCL). We use the contrastive learning strategy to pre-train the deep model on the large user-tagged data (referred to as weakly-labeled data) and then the pre-trained model is fine-tuned on the small human-annotated data. We refine the contrastive loss function to better exploit inter-class contrastive patterns, making contrastive learning more applicable to the weakly-supervised setting. Besides, multiple sampling on different sentiment pairs reduces the negative impact of label noises. SWCL captures the diverse sentiment semantics of weakly labeled data and improves their suitability for downstream sentiment classification tasks. Our method outperforms the other baseline methods in experiments on the Amazon review, Twitter, and SST-5 datasets. Even when fine-tuned on 0.5 percent of the training data (i.e. 32 instances), our framework significantly boosts the deep models’ performance, demonstrating its robustness in a few-shot learning scenario.

Stochastic surprisal: An inferential measurement of free energy in neural networks

“…For instance, training on gaussian blurred images does not guarantee a performance increase in motion blur images (Geirhos et al, 2018b ). Other proposed methods include training on style-transferred images (Geirhos et al, 2018a ), training on adversarial images (Hendrycks and Dietterich, 2019 ), training on simulated noisy virtual images (Temel et al, 2017 ), and self-supervised methods like SimCLR (Chen et al, 2020 ) that train by augmenting distortions. Augmix (Hendrycks et al, 2019 ) creates multiple chains of augmentations to train the base network.…”

“…In Table 3 , we compare the Top-1 accuracy between perception-only inference and our proposed stochastic surprisal-based inference. All the state-of-the-art techniques require additional training data—noisy images (Vasiljevic et al, 2016 ), adversarial images (Hendrycks and Dietterich, 2019 ), self-supervision SimCLR augmentations (Chen et al, 2020 ), and augmentation chains (Hendrycks et al, 2019 ). We term these perception-only techniques as f ′(·) and we actively infer on top of them.…”

Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective

“…Subsequently, we show that baking ARCO into contrastive pre-training (i.e., MONA [84]) provides an efficient pixel-wise contrastive learning paradigm to train deep networks that generalize well beyond training data. ARCO is easy to implement, being built on top of off-the-shelf pixellevel contrastive learning framework [5,21,37,41,69], and consistently improves overall segmentation quality across all label ratios and datasets.…”

“…Recently, a significant amount of research efforts [4,48,52,76,89,96,97] have resorted to unsupervised or semisupervised learning techniques for improving the segmentation robustness. One of the most effective methods is contrastive learning (CL) [21,39,41,59]. It aims to learn useful representations by contrasting semantically similar (positive) and dissimilar (negative) pairs of data points sampled from the massive unlabeled data.…”

“…Nevertheless, in practical clinical settings, the deployed CL models often ask for strong model robustness, i.e., being robust against the collapse problems whereby all representations collapse into constant features [5,21,23,41,71,90] or only span a lower-dimensional subspace [43,45,70,87]. Therefore, a more challenging requirement arises, which is far beyond the scope of segmentation quality especially for such safety-critical scenarios.…”