Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach

Abstract

Fine-tuned pre-trained language models (LMs) achieve enormous success in many natural language processing (NLP) tasks, but they still require excessive labeled data in the fine-tuning stage. We study the problem of fine-tuning pre-trained LMs using only weak supervision, without any labeled data. This problem is challenging because the high capacity of LMs makes them prone to overfitting the noisy labels generated by weak supervision. To address this problem, we develop a contrastive self-training framework, COSINE, to enable fine-tuning LMs with weak supervision. Underpinned by contrastive regularization and confidence-based reweighting, this contrastive self-training framework can gradually improve model fitting while effectively suppressing error propagation. Experiments on sequence, token, and sentence pair classification tasks show that our model outperforms the strongest baseline by large margins on 7 benchmarks in 6 tasks, and achieves competitive performance with fully-supervised fine-tuning methods.

Speech Detection Task Against Asian Hate: BERT the Central, While Data-Centric Studies the Crucial

“…Based on the study, future work may concentrate not only on data-centric research but also on the trade-off between performance and robustness considering classification based on small corpora to mitigate the effect of adversarial behavior. Additionally, future research could concentrate on how to fine-tune models with weak supervision: it could be expensive when collecting more data as well as label them all, and more studies have focused on the relevant contents (Yu et al, 2020;Awasthi et al, 2020).…”

ARS2: Adaptive Ranking-based Sample Selection for Weakly supervised Class-imbalanced Text Classification

“…Weak Supervision Methods: (1) COSINE (Yu et al, 2020) The COSINE method uses weakly labeled data to fine-tune pre-trained language models by contrastive self-training.…”

“…Weak Supervision. Weak supervision aims to reduce the cost of annotation, and has been widely applied to perform both classification (Ratner et al, 2016b(Ratner et al, , 2019aFu et al, 2020;Yu et al, 2020; and sequence tagging (Lison et al, 2020;Nguyen et al, 2017;Safranchik et al, 2020;Lan et al, 2020) to help reduce human labor required for annotation. Weak supervision builds on many previous approaches in machine learning, such as distant supervision (Mintz et al, 2009;Hoffmann et al, 2011;Takamatsu et al, 2012), crowdsourcing (Gao et al, 2011;Krishna et al, 2016), co-training methods (Blum and Mitchell, 1998), pattern-based supervision (Gupta and Manning, 2014), and feature annotation (Mann and McCallum, 2010;Zaidan and Eisner, 2008).…”

“…For COSINE (Yu et al, 2020), we search learning rate from [1 × 10 −5 , 3 × 10 −5 , 1 × 10 −6 , 3 × 10 −6 ], teacher model update frequency from [50,100,200], regularize power scale from [0.01, 0.05, 0.1] and margin threshold γ C from [0.1, 0.3, 0.5, 0.7, 0.9]. For Denoise , we search learning rate from [1 ×…”

“…Owing to this considerable cost, the weak supervision (WS) paradigm has increasingly been used to reduce human efforts (Ratner et al, 2016a;. This approach synthesizes training labels with labeling rules to significantly improve the efficiency of creating training sets and have achieved competitive results in natural language processing (NLP) (Yu et al, 2020;Rühling Cachay et al, 2021). However, existing methods leveraging the WS paradigm to perform NLP tasks mostly focus on reducing the noise in training labels brought by labeling rules, while ignoring the common and critical problem of data imbalance.…”