Computational Argumentation Quality Assessment in Natural Language

Abstract

Research on computational argumentation faces the problem of how to automatically assess the quality of an argument or argumentation. While different quality dimensions have been approached in natural language processing, a common understanding of argumentation quality is still missing. This paper presents the first holistic work on computational argumentation quality in natural language. We comprehensively survey the diverse existing theories and approaches to assess logical, rhetorical, and dialectical quality dimensions, and we derive a systematic taxonomy from these. In addition, we provide a corpus with 320 arguments, annotated for all 15 dimensions in the taxonomy. Our results establish a common ground for research on computational argumentation quality assessment.

Building an Argument Search Engine for the Web

“…However, as Table 1 exemplifies, we allow for model extensions to integrate most of them, such as the roles of Toulmin (1958) or the schemes of Walton et al (2008). Similarly, it is possible to add the various scores that can be computed for an argument, such as different quality ratings (Wachsmuth et al, 2017a). This way, they can still be employed in the assessment and ranking of arguments.…”

Argumentation Quality Assessment: Theory vs. Practice

“…Dialectical quality dimensions resemble those of cogency, but arguments are judged specifically by their reasonableness for achieving agreement (van Eemeren and Grootendorst, 2004). Wachsmuth et al (2017a) point out that dialectical builds on rhetorical, and rhetorical builds on logical quality. They derive a unifying taxonomy from the major theories, decomposing quality hierarchically into cogency, effectiveness, reasonableness, and subdimensions.…”

“…For Hypotheses 1 and 2, we consider all 736 pairs of arguments from Habernal and Gurevych (2016a) where both have been annotated by Wachsmuth et al (2017a). For each pair (A, B) with A being 1 Source code and annotated data: http://www.arguana.com more convincing than B, we check whether the ratings of A and B for each dimension (averaged over all annotators) show a concordant difference (i.e., a higher rating for A), a disconcordant difference (lower), or a tie.…”

"PageRank" for Argument Relevance

“…In (Wachsmuth et al, 2017), we comprehensively survey theories on argumentation quality as well as computational approaches to specific quality dimensions. Among the latter, Persing and Ng (2015) rely on manual annotations of essays to predict how strong an essay's argument is-a naturally subjective and non-scalable assessment.…”

“…In this regard, the proposed PageRank approach serves to retrieve relevant candidate arguments. These arguments should then be further assessed, e.g., in terms of the soundness of their inference or other quality dimensions (Wachsmuth et al, 2017). At web scale, however, our approach poses several challenges of processing natural language text, most of which refer to the construction of a reliable argument graph.…”