Automatic Argument Quality Assessment - New Datasets and Methods

Abstract

We explore the task of automatic assessment of argument quality. To that end, we actively collected 6.3k arguments, more than a factor of five compared to previously examined data. Each argument was explicitly and carefully annotated for its quality. In addition, 14k pairs of arguments were annotated independently, identifying the higher quality argument in each pair. In spite of the inherent subjective nature of the task, both annotation schemes led to surprisingly consistent results. We release the labeled datasets to the community. Furthermore, we suggest neural methods based on a recently released language model, for argument ranking as well as for argument-pair classification. In the former task, our results are comparable to state-of-the-art; in the latter task our results significantly outperform earlier methods. * These authors equally contributed to this work. 1 For more details:https://www.research. ibm.com/artificial-intelligence/ project-debater/live/

A Large-Scale Dataset for Argument Quality Ranking: Construction and Analysis

“…Arguments in their datasets, collected in the context of Speech by Crowd experiments, are suited to the use-case of civic engagement platforms, giving premium to the usability of an argument in oral communication. Our dataset differs in three respects: (1) our dataset is larger by a factor of 5 compared to previous datasets annotated for point-wise quality; (2) our data were collected mainly from crowd contributors that presumably better represent the general population compared to targeted audiences such as debate clubs; (3) we performed an extensive analysis of argument scoring methods and introduce superior scoring methods that consider annotators credibility without removing them entirely from the labeled data, as is done in Toledo et al (2019).…”

“…More recently, IBM also introduced Speech by Crowd, a service which supports the collection of free-text arguments from large audiences on debatable topics to generate meaningful narratives (Toledo et al 2019). An important sub-task of this service is automatic assessment of argu-ment quality, which is the focus of the present work.…”

“…Earlier work on assessing argument quality relied on a comparative pair-wise approach, aiming to identify the higher-quality argument within each pair of arguments (Habernal and Gurevych 2016;Simpson and Gurevych 2018). Recently Toledo et al (2019) proposed a pointwise argument quality prediction scheme, which scales linearly with data size. Correspondingly, here we focus on this paradigm which is clearly less demanding, especially in scenarios where many arguments should be considered.…”

“…A major contribution of this work is introducing a novel dataset of arguments, carefully annotated for point-wise quality, IBM-ArgQ-Rank-30kArgs, referred henceforth as IBM-Rank-30k. The dataset includes around 30k arguments, 5 times larger than the largest annotated point-wise data released to date (Toledo et al 2019). Similarly to Toledo et al (2019), arguments were collected actively -as opposed to being extracted from debate portals (Habernal and Gurevych 2016) -with strict length limitations, accompanied by extensive quality control measures.…”

“…The question remains, of how to extract a continuous quality score out of the binary annotations provided by many annotators. Toledo et al (2019) took a straightforward simpleaverage approach; here, we provide an extensive comparison between potential scoring functions, analyzing the differences between these models, and their impact on training and evaluating a learning algorithm.…”