Few-Shot Cross-Lingual Stance Detection with Sentiment-Based Pre-training

Abstract

The goal of stance detection is to determine the viewpoint expressed in a piece of text towards a target. These viewpoints or contexts are often expressed in many different languages depending on the user and the platform, which can be a local news outlet, a social media platform, a news forum, etc. Most research on stance detection, however, has been limited to working with a single language and on a few limited targets, with little work on cross-lingual stance detection. Moreover, non-English sources of labelled data are often scarce and present additional challenges. Recently, large multilingual language models have substantially improved the performance on many non-English tasks, especially such with a limited number of examples. This highlights the importance of model pre-training and its ability to learn from few examples. In this paper, we present the most comprehensive study of cross-lingual stance detection to date: we experiment with 15 diverse datasets in 12 languages from 6 language families, and with 6 low-resource evaluation settings each. For our experiments, we build on pattern-exploiting training (PET), proposing the addition of a novel label encoder to simplify the verbalisation procedure. We further propose sentiment-based generation of stance data for pre-training, which shows sizeable improvement of more than 6% F1 absolute in few-shot learning settings compared to several strong baselines.

Improved Multi-label Classification under Temporal Concept Drift: Rethinking Group-Robust Algorithms in a Label-Wise Setting

“…Further on, we would like to investigate the use of zero-shot LWAN methods (Rios and Kavuluru, 2018;Chalkidis et al, 2020a), which currently harm averaged performance in favor of improved worst case performance. Label encodings based on contextualized word representations generated by pre-trained language models (Hardalov et al, 2021) may mitigate the effect of using non-contextualized ones (e.g., Word2Vec).…”

A Survey on Stance Detection for Mis- and Disinformation Identification

“…For Arabic, Khouja (2020) achieved 76.7 F1 for stance detection on the ANS dataset using mBERT. Similarly, Hardalov et al (2022) applied pattern-exploiting training (PET) with sentiment pre-training in a cross-lingual setting showing sizeable improvements on 15 datasets. Alhindi et al ( 2021) showed that language-specific pre-training was pivotal, outperforming the state of the art on AraStance (52 F1) and Arabic FC (78 F1).…”

“…They further integrated label embeddings (Augenstein et al, 2018), and eventually developed an end-toend unsupervised framework for predicting stance from a set of unseen target labels. Hardalov et al (2022) explored PET (Schick and Schütze, 2021) in a cross-lingual setting, combining datasets with different label inventories by modelling the task as a cloze question answering one. They showed that MDL helps for low-resource and substantively for full-resource scenarios.…”

“…This can lead to a significant disparity for label performance (see Section 4). Several techniques have been proposed to mitigate this, such as sampling strategies (Nie et al, 2019), weighting classes (Veyseh et al, 2017), 3 crafting artificial examples from auxiliary tasks Hardalov et al, 2022), or training on multiple datasets (Schiller et al, 2021;Hardalov et al, 2021Hardalov et al, , 2022.…”

“…Currently, only a handful of datasets for factuality and stance cover languages other than English (see Table 1), and they are small in size and do not offer a cross-lingual setup. Recently, Vamvas and Sennrich (2020) proposed such a setup for three languages for stance in debates, Schick and Schütze (2021) explored few-shot learning, and Hardalov et al (2022) extended that paradigm with sentiment and stance pre-training and evaluated on twelve languages from various domains. Since cultural norms and expressed linguistic phenomena play a crucial role in understanding the context of a claim (Sap et al, 2019), we do not argue for a completely language-agnostic framework.…”