LaDiff ULMFiT: A Layer Differentiated Training Approach for ULMFiT

Abstract

In our paper we present Deep Learning models with a layer differentiated training method which were used for the SHARED TASK @ CONSTRAINT 2021 sub-tasks COVID19 Fake News Detection in English and Hostile Post Detection in Hindi. We propose a Layer Differentiated training procedure for training a pre-trained ULMFiT [8] model. We used special tokens to annotate specific parts of the tweets to improve language understanding and gain insights on the model making the tweets more interpretable. The other two submissions included a modified RoBERTa model and a simple Random Forest Classifier. The proposed approach scored a precision and f1-score of 0.96728972 and 0.967324832 respectively for sub-task COVID19 Fake News Detection in English. Also, Coarse Grained Hostility f1 Score and Weighted Fine Grained f1 score of 0.908648 and 0.533907 respectively for sub-task Hostile Post Detection in Hindi. The proposed approach ranked 61st out of 164 in the sub-task "COVID19 Fake News Detection in English" and 18th out of 45 in the sub-task "Hostile Post Detection in Hindi".

Combating the infodemic: COVID-19 induced fake news recognition in social media networks

“…1, and the pseudocode for the same is given in Algorithm 1. All these models were built using Python's sci-kit-learn 4 and transformer library 5 .…”

“…Output: Decision in the form of value. 4 https://scikit-learn.org/stable/. In Model 2, multiple RNN variants such as LSTM, BiL-STM, GRU, and BiGRU with two stacked layers of 60 and 30 neurons were developed on the features extracted from BERT embeddings.…”

Transfer learning and GRU-CRF augmentation for Covid-19 fake news detection

“…The social engagements on articles can be a significant feature for fake news detection (to find the semantic relationship between news articles and writers) [23]. In the Fake News Detection research field, many datasets can be used, such as PolitiFact [24,25], Fake News Kaggle [18,26], The Fake News Challenge (FNC-1) [27,28], and Constraint@AAAI2021 -COVID19 Fake News Detection [9,10,12,11]. Ahmad et al [18] developed Fake News Detection Using Machine Learning Ensemble Methods consists of Logistic Regression, Support Vector Machine, Random Forest (RF), etc.…”

“…There are two main concepts: An Inconsistency Graph and Energy Flow. Azhan et al [10] proposed a Layer Differentiated training procedure for training a pretrained ULMFiT model. They also used unique tokens to annotate specific parts of the tweets to improve language understanding and gain insights into the model, making the tweets more interpretable.…”

“…Several previous studies have contributed to this Constraint @ AAAI2021 -COVID19 Fake News Detection in English shared task. Azhan et al [10] apply a Layer Differentiated training procedure for training a pre-trained ULMFiT, Kakwani et al [11] compile the IndicGLUE benchmark for language, Baris et al [12] propose a modeling framework for those features by using BERT language model and external sources. Considering the number of researches utilizing the dataset, we think it crucial to contribute to this shared-task by using another approach.…”