Predicting Argument Density from Multiple Annotations

Rocha, Gil; Leite, Bernardo; Trigo, Luís; Cardoso, Henrique Lopes; Sousa‐Silva, Rui; Carvalho, Paula; Martins, Bruno; Won, Miguel

doi:10.1007/978-3-031-08473-7_21

Cited by 2 publications

(3 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While other works that simply partition texts into argument and non argument types might achieve even higher IAA, our task is at a higher difficulty level. Compared to similar efforts annotating refined argument structures (Rocha et al, 2022) ( u α = 0.33), our IAA score is significantly improved.…”

Section: Annotation Rounds All Annotations Were Donementioning

confidence: 91%

Automatic Analysis of Substantiation in Scientific Peer Reviews

Guo,

Shang,

Rennard

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

With the increasing amount of problematic peer reviews in top AI conferences, the community is urgently in need of automatic quality control measures. In this paper, we restrict our attention to substantiation -one popular quality aspect indicating whether the claims in a review are sufficiently supported by evidence -and provide a solution automatizing this evaluation process. To achieve this goal, we first formulate the problem as claim-evidence pair extraction in scientific peer reviews, and collect Substan-Review, the first annotated dataset for this task. SubstanReview consists of 550 reviews from NLP conferences annotated by domain experts. On the basis of this dataset, we train an argument mining system to automatically analyze the level of substantiation in peer reviews. We also perform data analysis on the SubstanReview dataset to obtain meaningful insights on peer reviewing quality in NLP conferences over recent years. The dataset is available at https: //github.com/YanzhuGuo/SubstanReview.

show abstract

Section: Annotation Rounds All Annotations Were Donementioning

confidence: 91%

Automatic Analysis of Substantiation in Scientific Peer Reviews

Guo,

Shang,

Rennard

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

show abstract

“…In both cases, argumentation annotations are kept, either manually or automatically projected from the original annotated corpus. Rocha et al (2022b) have made some inroads into applying Freeman's model to a less structured argumentative genre-opinion articles as published in a Portuguese newspaper. In their approach, each of a set of 373 articles has been annotated considering the full Freeman model, including several kinds of argument structures (linked, convergent, divergent, and serial) as well as two kinds of relations (support and attack).…”

Section: Freeman Annotationsmentioning

confidence: 99%

“…However, for the argumentative units in which there is agreement, higher-level component analysis (such as types and roles of propositions, and macro-structure of arguments) can obtain considerable agreement. Based on this corpus, they have explored the recent trend into perspectivist approaches to NLP (Basile et al 2021), by considering different ways of aggregating annotations (Rocha et al 2022a).…”

Section: Freeman Annotationsmentioning

confidence: 99%

Argumentation models and their use in corpus annotation: Practice, prospects, and challenges

et al. 2023

Self Cite

View full text Add to dashboard Cite

The study of argumentation is transversal to several research domains, from philosophy to linguistics, from the law to computer science and artificial intelligence. In discourse analysis, several distinct models have been proposed to harness argumentation, each with a different focus or aim. To analyze the use of argumentation in natural language, several corpora annotation efforts have been carried out, with a more or less explicit grounding on one of such theoretical argumentation models. In fact, given the recent growing interest in argument mining applications, argument-annotated corpora are crucial to train machine learning models in a supervised way. However, the proliferation of such corpora has led to a wide disparity in the granularity of the argument annotations employed. In this paper, we review the most relevant theoretical argumentation models, after which we survey argument annotation projects closely following those theoretical models. We also highlight the main simplifications that are often introduced in practice. Furthermore, we glimpse other annotation efforts that are not so theoretically grounded but instead follow a shallower approach. It turns out that most argument annotation projects make their own assumptions and simplifications, both in terms of the textual genre they focus on and in terms of adapting the adopted theoretical argumentation model for their own agenda. Issues of compatibility among argument-annotated corpora are discussed by looking at the problem from a syntactical, semantic, and practical perspective. Finally, we discuss current and prospective applications of models that take advantage of argument-annotated corpora.

show abstract

Predicting Argument Density from Multiple Annotations

Cited by 2 publications

References 22 publications

Automatic Analysis of Substantiation in Scientific Peer Reviews

Automatic Analysis of Substantiation in Scientific Peer Reviews

Argumentation models and their use in corpus annotation: Practice, prospects, and challenges

Contact Info

Product

Resources

About