Ranking Passages for Argument Convincingness

Abstract

In data ranking applications, pairwise annotation is often more consistent than cardinal annotation for learning ranking models. We examine this in a case study on ranking text passages for argument convincingness. Our task is to choose text passages that provide the highest-quality, most-convincing arguments for opposing sides of a topic. Using data from a deployed system within the Bing search engine, we construct a pairwiselabeled dataset for argument convincingness that is substantially more comprehensive in topical coverage compared to existing public resources. We detail the process of extracting topical passages for queries submitted to a search engine, creating annotated sets of passages aligned to different stances on a topic, and assessing argument convincingness of passages using pairwise annotation. Using a state-of-the-art convincingness model, we evaluate several methods for using pairwiseannotated data examples to train models for ranking passages. Our results show pairwise training outperforms training that regresses to a target score for each passage. Our results also show a simple 'win-rate' score is a better regression target than the previously proposed page-rank target. Lastly, addressing the need to filter noisy crowd-sourced annotations when constructing a dataset, we show that filtering for transitivity within pairwise annotations is more effective than filtering based on annotation confidence measures for individual examples.

A Large-Scale Dataset for Argument Quality Ranking: Construction and Analysis

“…Following Simpson and Gurevych (2018), we report average Pearson (r) and Spearman (ρ) correlations, and compare the results of our methods to the Bi-LSTM and GPPL methods published there, as well as to the EviConvNet method, the best result from Gleize et al (2019), and to the SWE+FFNN method, the best result from Potash, Ferguson, and Hazen (2019).…”

“…An alternative approach to assess arguments is to focus on their relative convincingness, by comparing pairs of arguments with similar stance. This approach is introduced in Habernal and Gurevych; Simpson and Gurevych (2016;, and further assessed in Potash, Bhattacharya, and Rumshisky (2017), Gleize et al (2019), and Potash, Ferguson, and Hazen (2019). As part of their work, Habernal and Gurevych (2016) introduce two datasets: UKPConvAr-gRank (henceforth, UKPRank) and UKPConvArgAll, which contain 1k and 16k arguments and argument-pairs, respectively.…”

“…Their solution is based on a Siamese neural network, which outperforms the results achieved in Simpson and Gurevych (2018) on the UKP datasets, as well as several baselines on their own dataset, IBM-ConvEnv. Potash, Bhattacharya, and Rumshisky (2017) present a method that is based on representing an argument by the sum of its token embeddings, extended in Potash, Ferguson, and Hazen (2019) to include a Feed Forward Neural Network. These two works outperform Simpson and Gurevych (2018) on the UKP datasets for both the pair-wise and point-wise tasks, respectively.…”

Beyond facts – a survey and conceptualisation of claims in online discourse analysis

“…They can be expressed explicitly or implicitly [146]. Fragments can be messages such as tweets or posts [55,86], paragraphs [144] or complete articles [70]. Joseph et al [86] see stances as latent properties of users rather than text fragments.…”

Learning From Revisions: Quality Assessment of Claims in Argumentation at Scale

“…The more convincing argument is then predicted using a feature-rich SVM and a simple bidirectional LSTM. Other approaches to the same task map passage representations to real-valued scores using Gaussian Process Preference Learning (Simpson and Gurevych, 2018) or represent arguments by the sum of their token embeddings (Potash et al, 2017), later extended by a Feed Forward Neural Network (Potash et al, 2019). Recently, Gleize et al (2019) employed a Siamese neural network to rank arguments by the convincingness of evidence.…”