The CASS Technique for Evaluating the Performance of Argument Mining

Duthie, Rory; Lawrence, John; Budzyńska, Katarzyna; Reed, Chris

doi:10.18653/v1/w16-2805

Cited by 9 publications

(10 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The full annotation guidelines have been validated by calculating the inter-annotator agreement on a 11.3% sample, resulting in a Cohen (1960)'s of 0.610, and a CASS (Duthie et al 2016) of 0.752-both indicating substantial agreement according to Landis and Koch (1977)'s standard interpretation of the kappa metric. The resulting annotated US2016 corpus is freely available online at http://corpo ra.aifdb .org/ US201 6.…”

Section: The Us2016 Corpusmentioning

confidence: 99%

Annotating Argument Schemes

et al. 2020

Self Cite

View full text Add to dashboard Cite

Argument schemes are abstractions substantiating the inferential connection between premise(s) and conclusion in argumentative communication. Identifying such conventional patterns of reasoning is essential to the interpretation and evaluation of argumentation. Whether studying argumentation from a theory-driven or data-driven perspective, insight into the actual use of argumentation in communicative practice is essential. Large and reliably annotated corpora of argumentative discourse to quantitatively provide such insight are few and far between. This is all the more true for argument scheme corpora, which tend to suffer from a combination of limited size, poor validation, and the use of ad hoc restricted typologies. In the current paper, we describe the annotation of schemes on the basis of two distinct classifications: Walton's taxonomy of argument schemes, and Wagemans' Periodic Table of Arguments. We describe the annotation procedure for each, and the quantitative characteristics of the resulting annotated text corpora. In doing so, we extend the annotation of the preexisting US2016 corpus of televised election debates, resulting in, to the best of our knowledge, the two largest consistently annotated corpora of schemes in argumentative dialogue publicly available. Based on evaluation in terms of inter-annotator agreement, we propose further improvements to the guidelines for annotating schemes: the argument scheme key, and the Argument Type Identification Procedure.

show abstract

Section: The Us2016 Corpusmentioning

confidence: 99%

Annotating Argument Schemes

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…While we prefer Cohen's j metric over percentage agreement, because it accounts for chance agreements between annotators, it has the drawback that errors can be passed from text segmentation-a non-fixed task-to identifying relations, thus not providing a comprehensive agreement score. Duthie et al (2016b) introduce the Combined Argument Similarity Score j (CASS-j) aimed at overcoming this by calculating intermediate agreement scores for the composite tasks of text segmentation, annotation of dialogical relations, and annotation of propositional relations, before combining the detailed calculations into an overall CASS-j score (while still accounting for chance agreements.) In Table 2, we include the pairwise CASS-j scores, resulting in an overall CASS-j for US2016 of 0.752.…”

Section: Validationmentioning

confidence: 99%

Argumentation in the 2016 US presidential elections: annotated corpora of television debates and social media reaction

Visser

Konat

Duthie

et al. 2019

Lang Resources & Evaluation

Self Cite

View full text Add to dashboard Cite

In this paper we present US2016, the largest publicly available set of corpora of annotated dialogical argumentation. The annotation covers argumentative relations, dialogue acts and pragmatic features. The corpora comprise transcriptions of television debates leading up to the 2016 US presidential elections, and reactions to the debates on Reddit. These two constitutive parts of the corpora are integrated by means of the intertextual correspondence between them. The rhetorical richness and high argument density of the communicative context results in cross-genre corpora that are robust resources for the study of the dialogical dynamics of argumentation in three ways: first, in empirical strands of research in discourse analysis and argumentation studies; second, in the burgeoning field of argument mining where automatic techniques require such data; and third, in formulating algorithmic techniques for sensemaking through the development of Argument Analytics.

show abstract

“…For the inter-annotator agreement, a systematic sample of 10% of the corpus was extracted (by selecting every 10th argument map from the corpus) and annotated by the second annotator. The samples were compared using the Combined Argument Similarity Score (CASS) technique ( [Duthie et al 2016], resulting in Cohen's κ = 0.92. 8 The number of occurrences for each label is given in Table II.…”

Section: The Erulemaking Debate Corpusmentioning

confidence: 99%

Using Argumentative Structure to Interpret Debates in Online Deliberative Democracy and eRulemaking

Lawrence

Park

Budzyńska

et al. 2017

ACM Trans. Internet Technol.

Self Cite

View full text Add to dashboard Cite

Governments around the world are increasingly utilising online platforms and social media to engage with, and ascertain the opinions of, their citizens. Whilst policy makers could potentially benefit from such enormous feedback from society, they first face the challenge of making sense out of the large volumes of data produced. In this article, we show how the analysis of argumentative and dialogical structures allows for the principled identification of those issues that are central, controversial, or popular in an online corpus of debates. Although areas such as controversy mining work towards identifying issues that are a source of disagreement, by looking at the deeper argumentative structure, we show that a much richer understanding can be obtained. We provide results from using a pipeline of argument-mining techniques on the debate corpus, showing that the accuracy obtained is sufficient to automatically identify those issues that are key to the discussion, attracting proportionately more support than others, and those that are divisive, attracting proportionately more conflicting viewpoints.

show abstract

The CASS Technique for Evaluating the Performance of Argument Mining

Cited by 9 publications

References 25 publications

Annotating Argument Schemes

Annotating Argument Schemes

Argumentation in the 2016 US presidential elections: annotated corpora of television debates and social media reaction

Using Argumentative Structure to Interpret Debates in Online Deliberative Democracy and eRulemaking

Contact Info

Product

Resources

About