Agreement Prediction of Arguments in Cyber Argumentation for Detecting Stance Polarity and Intensity

Sirrianni, Joseph; Liu, Xiaoqing; Adams, Douglas

doi:10.18653/v1/2020.acl-main.509

Cited by 12 publications

(7 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There also exists a subfield of StD that specializes in classifying the stance towards a given rumour [14,55]. In addition to stance labels, Sirrianni et al [39] also predict the intensity of a stance towards a post.…”

Section: Related Workmentioning

confidence: 99%

Stance Detection Benchmark: How Robust is Your Stance Detection?

2021

View full text Add to dashboard Cite

Stance detection (StD) aims to detect an author’s stance towards a certain topic and has become a key component in applications like fake news detection, claim validation, or argument search. However, while stance is easily detected by humans, machine learning (ML) models are clearly falling short of this task. Given the major differences in dataset sizes and framing of StD (e.g. number of classes and inputs), ML models trained on a single dataset usually generalize poorly to other domains. Hence, we introduce a StD benchmark that allows to compare ML models against a wide variety of heterogeneous StD datasets to evaluate them for generalizability and robustness. Moreover, the framework is designed for easy integration of new datasets and probing methods for robustness. Amongst several baseline models, we define a model that learns from all ten StD datasets of various domains in a multi-dataset learning (MDL) setting and present new state-of-the-art results on five of the datasets. Yet, the models still perform well below human capabilities and even simple perturbations of the original test samples (adversarial attacks) severely hurt the performance of MDL models. Deeper investigation suggests overfitting on dataset biases as the main reason for the decreased robustness. Our analysis emphasizes the need of focus on robustness and de-biasing strategies in multi-task learning approaches. To foster research on this important topic, we release the dataset splits, code, and fine-tuned weights.

show abstract

Section: Related Workmentioning

confidence: 99%

Stance Detection Benchmark: How Robust is Your Stance Detection?

2021

View full text Add to dashboard Cite

show abstract

“…Most available datasets for stance detection only indicate the polarity of the stance against a target (in favor, against, neutral) (Mohammad et al, 2016a;Stab et al, 2018), however recent work has aimed to annotate the stance intensity as well. Sirrianni et al (2020) introduced a dataset for agreement prediction where the aim is to predict a post's stance towards its parent post in a 5-point scale and Levow et al (2014) present a corpus of spontaneous, conversational speech with stance annotations on a 7-point scale. Both datasets contain statement-response pairs from conversational settings and are more suitable for detecting whether two topicindependent statements are in agreement, rather than detecting a stance from one statement on a sociopolitical issue.…”

Section: Related Workmentioning

confidence: 99%

Investigating User Radicalization: A Novel Dataset for Identifying Fine-Grained Temporal Shifts in Opinion

Sakketou¹,

Lahnala²,

Vogel³

et al. 2022

Preprint

View full text Add to dashboard Cite

There is an increasing need for the ability to model fine-grained opinion shifts of social media users, as concerns about the potential polarizing social effects increase. However, the lack of publicly available datasets that are suitable for the task presents a major challenge. In this paper, we introduce an innovative annotated dataset for modeling subtle opinion fluctuations and detecting fine-grained stances. The dataset includes a sufficient amount of stance polarity and intensity labels per user over time and within entire conversational threads, thus making subtle opinion fluctuations detectable both in long term and in short term. All posts are annotated by non-experts and a significant portion of the data is also annotated by experts. We provide a strategy for recruiting suitable non-experts. Our analysis of the inter-annotator agreements shows that the resulting annotations obtained from the majority vote of the non-experts are of comparable quality to the annotations of the experts. We provide analyses of the stance evolution in short term and long term levels, a comparison of language usage between users with vacillating and resolute attitudes, and fine-grained stance detection baselines.

show abstract

“…Stance classification has been a subject of research in many different environments, such as congressional debates [21], online debates on social media [19,4] and company-internal discussions [14]. Previous approaches focus on learning topic-specific models to classify stances of related claims with machine learning models [2,10,18] as well as deep learning models [20,7,8,15,16,24]. Previous work has also looked at doing stance classification at challenging situations such as zero-shot [1] and unsupervised settings [19,9,11].…”

Section: Related Workmentioning

confidence: 99%

Distant finetuning with discourse relations for stance classification

Jin¹,

Xu²,

Song³

et al. 2022

Preprint

View full text Add to dashboard Cite

Approaches for the stance classification task, an important task for understanding argumentation in debates and detecting fake news, have been relying on models which deal with individual debate topics. In this paper, in order to train a system independent from topics, we propose a new method to extract data with silver labels from raw text to finetune a model for stance classification. The extraction relies on specific discourse relation information, which is shown as a reliable and accurate source for providing stance information. We also propose a 3-stage training framework where the noisy level in the data used for finetuning decreases over different stages going from the most noisy to the least noisy. Detailed experiments show that the automatically annotated dataset as well as the 3-stage training help improve model performance in stance classification. Our approach ranks 1 st among 26 competing teams in the stance classification track of the NLPCC 2021 shared task Argumentative Text Understanding for AI Debater, which confirms the effectiveness of our approach.

show abstract

Agreement Prediction of Arguments in Cyber Argumentation for Detecting Stance Polarity and Intensity

Cited by 12 publications

References 43 publications

Stance Detection Benchmark: How Robust is Your Stance Detection?

Stance Detection Benchmark: How Robust is Your Stance Detection?

Investigating User Radicalization: A Novel Dataset for Identifying Fine-Grained Temporal Shifts in Opinion

Distant finetuning with discourse relations for stance classification

Contact Info

Product

Resources

About