Diversify Your Datasets: Analyzing Generalization via Controlled Variance in Adversarial Datasets

Rozen, Ohad; Shwartz, Vered; Aharoni, Roee; Dagan, Ido

doi:10.48550/arxiv.1910.09302

Cited by 1 publication

(1 citation statement)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1. predicates and names are sampled from different, disjunct domains (texts are about, e.g., allergies and family relations versus, e.g., badminton and cooking) to test a model's robustness to lexical diversity (Rozen et al, 2019); 2. similarly, AAAC01 applies automatic paraphrasing (Alisetti, 2021) to the final source text whereas AAAC02 doesn't; 3. AAAC02 allows for imprecise renditions of logical formulas, while AAAC01 sticks to plain formulations to test robustness to variations in description of rules.…”

Section: Datasetsmentioning

confidence: 99%

DeepA2: A Modular Framework for Deep Argument Analysis with Pretrained Neural Text2Text Language Models

Betz¹,

Richardson²

2022

Proceedings of the 11th Joint Conference on Lexical and Computational Semantics

View full text Add to dashboard Cite

In this paper, we present and implement a multidimensional, modular framework for performing deep argument analysis (DeepA2) using current pre-trained language models (PTLMs). ArgumentAnalyst -a T5 model (Raffel et al., 2020) set up and trained within DeepA2 -reconstructs argumentative texts, which advance an informal argumentation, as valid arguments: It inserts, e.g., missing premises and conclusions, formalizes inferences, and coherently links the logical reconstruction to the source text. We create a synthetic corpus for deep argument analysis, and evaluate ArgumentAnalyst on this new dataset as well as on existing data, specifically EntailmentBank (Dalvi et al., 2021). Our empirical findings vindicate the overall framework and highlight the advantages of a modular design, in particular its ability to emulate established heuristics (such as hermeneutic cycles), to explore the model's uncertainty, to cope with the plurality of correct solutions (underdetermination), and to exploit higher-order evidence.

show abstract

Section: Datasetsmentioning

confidence: 99%