Semi-Supervised Text Classification via Self-Pretraining

Abstract

We present a neural semi-supervised learning model termed Self-Pretraining. Our model is inspired by the classic self-training algorithm. However, as opposed to self-training, Self-Pretraining is threshold-free, it can potentially update its belief about previously labeled documents, and can cope with the semantic drift problem. Self-Pretraining is iterative and consists of two classifiers.In each iteration, one classifier draws a random set of unlabeled documents and labels them. This set is used to initialize the second classifier, to be further trained by the set of labeled documents. The algorithm proceeds to the next iteration and the classifiers' roles are reversed. To improve the flow of information across the iterations and also to cope with the semantic drift problem, Self-Pretraining employs an iterative distillation process, transfers hypotheses across the iterations, utilizes a two-stage training model, uses an efficient learning rate schedule, and employs a pseudo-label transformation heuristic. We have evaluated our model in three publicly available social media datasets. Our experiments show that Self-Pretraining outperforms the existing state-of-the-art semisupervised classifiers across multiple settings. Our code is available at https://github.com/p-karisani/self_pretraining.

A new semi-supervised inductive transfer learning framework: Co-Transfer

“…For semi-supervised learning (Zhu, 2008;Van Engelen & Hoos, 2020;Karisani & Karisani, 2021), the main idea is to utilize a small number of labeled samples and a large amount of unlabeled samples to improve the performance of the learned hypothesis. One of the assumptions for semi-supervised learning is that the unlabeled examples hold the same distribution as that held by the labeled ones.…”

A chronological and geographical analysis of personal reports of COVID-19 on Twitter from the UK

“… 3 Initiatives relying on self-reporting of symptoms have included apps such as ‘COVID 19 symptom tracker’ ( https://covid.joinzoe.com/ ) in the UK, surveys disseminated via Facebook ( https://jpsm.umd.edu/research/facebook-%28covid%29-symptom-survey ), and analysis of posts on social media. 4 – 7 These initiatives may all provide useful complementary data to help populate modelling predictions of COVID-19 transmissions.…”

“…A social media mining approach has already been applied to tweets from users in the United States, China and Italy results. 4 – 8 This research has either simply used Twitter volume on COVID-19, or relied on the terms ‘cough’ and ‘fever’ or used synonyms for ‘COVID-19’ as a predictor with surprisingly good results. 4 – 8 …”

“…4 – 8 This research has either simply used Twitter volume on COVID-19, or relied on the terms ‘cough’ and ‘fever’ or used synonyms for ‘COVID-19’ as a predictor with surprisingly good results. 4 – 8 …”

View Distillation with Unlabeled Data for Extracting Adverse Drug Effects from User-Generated Data

“…We use unlabeled data to transfer the knowledge from the classifier in each view to the classifier in the other view. Additionally, we use a finetuning technique to mitigate the impact of noisy pseudo-labels after the initialization (Karisani and Karisani, 2021). As straightforward as it is to implement, our model achieves the state-of-the-art performance in the largest publicly available ADR dataset, i.e., SMM4H dataset.…”