SheffieldVeraAI at SemEval-2023 Task 3: Mono and Multilingual Approaches for News Genre, Topic and Persuasion Technique Classification

Abstract

This paper describes our approach for SemEval-2023 Task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multilingual setup. For Subtask 1 (News Genre), we propose an ensemble of fully trained and adapter mBERT models which was ranked joint-first for German, and had the highest mean rank of multi-language teams. For Subtask 2 (Framing), we achieved first place in 3 languages, and the best average rank across all the languages, by using two separate ensembles: a monolingual RoBERTa-MUPPET LARGE and an ensemble of XLM-RoBERTa LARGE with adapters and task adaptive pretraining. For Subtask 3 (Persuasion Techniques), we trained a monolingual RoBERTa-Base model for English and a multilingual mBERT model for the remaining languages, which achieved top 10 for all languages, including 2nd for English. For each subtask, we compared monolingual and multilingual approaches, and considered class imbalance techniques. 1

Comparison between parameter-efficient techniques and full fine-tuning: A case study on multilingual news article classification

“…For sub-task 1, the articles that are longer than 512 tokens are separated into sentences, which are then sampled sequentially from the beginning and the end of the article, preserving the original order, until the maximum of 512 tokens is reached. Such a truncation approach is motivated by our experiments on sub-task 1 data during the competition stage of SemEval 2023 Task 3 [16]. This approach yielded a significant improvement in the F1 macro score over the setting that…”

“…It should be noted that three of the systems that participated in the original SemEval-2023 Task 3 [15] evaluation exercise used adapters. Teams HHU [29] and NAP [30] entered only sub-task 3, in which they used adapters, whereas SheffieldVeraAI [16] applied adapters to sub-tasks 1 and 2. Initial performance analysis in these sub-tasks showed the effect of adapters to be inconsistent across the different sub-tasks.…”

“…Importantly, for sub-task 3 experiments, we include sentences that do not have assigned labels into the training data by assigning them a vector of zeros to indicate that they do not belong to any class. This approach was shown to significantly improve classification performance on this sub-task in our initial experiments [16,17] The multilabel sub-tasks 2 and 3 use confidence thresholds of 50% and 30%, respectively, after applying a sigmoid activation function to the logits. The confidence threshold for subtask 3 is purposefully lower and was selected according to our previous experiments [17], which revealed that its careful calibration can significantly influence the performance of the model.…”

HTE at ArAIEval Shared Task: Integrating Content Type Information in Binary Persuasive Technique Detection

“…Some works relied solely on real-world data to fine-tune a pre-trained language model for propaganda and persuasive technique detection (Costa et al, 2023), whereas others combined it with synthetically augmented data (Hasanain et al, 2023b). The ensemble approach was also investigated, in which various combined pre-trained language models are fine-tuned in a vanilla setting (Purificato and Navigli, 2023), or by using adapters (Wu et al, 2023).…”