Explanations for CommonsenseQA: New Dataset and Models

Aggarwal, Shourya; Mandowara, Divyanshu; Agrawal, Vishwajeet; Khandelwal, Dinesh; Singla, Parag; Garg, Dinesh

doi:10.18653/v1/2021.acl-long.238

Cited by 26 publications

(40 citation statements)

References 34 publications

(43 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Datasets in FEB To identify available datasets suitable for few-shot self-rationalization, we start with a recent overview of datasets with free-text explanations ) and filter them according to the following criteria: (i) the input is textual, (ii) the explanation consists of one sentence or 2-3 simple sentences, (iii) the task has a fixed set of possible labels, (iv) the explanation is Training sets for all classification tasks are balanced and contain 48 instances. Sources: E-SNLI (Camburu et al, 2018), ECQA (Aggarwal et al, 2021), COMVE , SBIC (Sap et al, 2020).…”

Section: Feb Benchmarkmentioning

confidence: 99%

Few-Shot Self-Rationalization with Natural Language Prompts

Marasović¹,

Beltagy²,

Downey³

et al. 2022

Findings of the Association for Computational Linguistics: NAACL 2022

View full text Add to dashboard Cite

Self-rationalization models that predict task labels and generate free-text elaborations for their predictions could enable more intuitive interaction with NLP systems. These models are, however, currently trained with a large amount of human-written free-text explanations for each task which hinders their broader usage. We propose to study a more realistic setting of self-rationalization using few training examples. We present FEB-a standardized collection of four existing English-language datasets and associated metrics. We identify the right prompting approach by extensively exploring natural language prompts on FEB. Then, by using this prompt and scaling the model size, we demonstrate that making progress on few-shot self-rationalization is possible. We show there is still ample room for improvement in this task: the average plausibility of generated explanations assessed by human annotators is at most 51% (with GPT-3), while plausibility of human explanations is 76%. We hope that FEB and our proposed approach will spur the community to take on the few-shot self-rationalization challenge.

show abstract

Section: Feb Benchmarkmentioning

confidence: 99%

Few-Shot Self-Rationalization with Natural Language Prompts

Marasović¹,

Beltagy²,

Downey³

et al. 2022

Findings of the Association for Computational Linguistics: NAACL 2022

View full text Add to dashboard Cite

show abstract

“…This issue could be alleviated by using model-independent criteria to categorize information content. For example, Aggarwal et al (2021) propose to quantify the information contained in a free-text explanation by calculating the number of distinct words (nouns, verbs, adjectives, and adverbs) per explanation.…”

Section: Information Contentmentioning

confidence: 99%

A survey on improving NLP models with human explanations

Hartmann¹,

Sonntag²

2022

Preprint

View full text Add to dashboard Cite

Training a model with access to human explanations can improve data efficiency and model performance on in-and out-of-domain data. Adding to these empirical findings, similarity with the process of human learning makes learning from explanations a promising way to establish a fruitful human-machine interaction. Several methods have been proposed for improving natural language processing (NLP) models with human explanations, that rely on different explanation types and mechanism for integrating these explanations into the learning process. These methods are rarely compared with each other, making it hard for practitioners to choose the best combination of explanation type and integration mechanism for a specific use-case. In this paper, we give an overview of different methods for learning from human explanations, and discuss different factors that can inform the decision of which method to choose for a specific usecase.

show abstract

“…Com-monsenseQA (Talmor et al, 2019) is a multiple choice task posed over commonsense questions. Crowdsourced free-text explanations for instances in CommonsenseQA are provided in the CoS-E v1.11 (Rajani et al, 2019) and ECQA (Aggarwal et al, 2021) datasets. ECQA explanations are counterfactual, i.e., annotators were instructed to explain not only the correct answer choice but also why the others are incorrect.…”

Section: Introductionmentioning

confidence: 99%

Reframing Human-AI Collaboration for Generating Free-Text Explanations

Wiegreffe¹,

Hessel²,

Swayamdipta³

et al. 2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Large language models are increasingly capable of generating fluent-appearing text with relatively little task-specific supervision. But can these models accurately explain classification decisions? We consider the task of generating free-text explanations using human-written examples in a few-shot manner. We find that (1) authoring higher quality prompts results in higher quality generations; and (2) surprisingly, in a head-to-head comparison, crowdworkers often prefer explanations generated by GPT-3 to crowdsourced explanations in existing datasets. Our human studies also show, however, that while models often produce factual, grammatical, and sufficient explanations, they have room to improve along axes such as providing novel information and supporting the label. We create a pipeline that combines GPT-3 with a supervised filter that incorporates binary acceptability judgments from humans in the loop. Despite the intrinsic subjectivity of acceptability judgments, we demonstrate that acceptability is partially correlated with various fine-grained attributes of explanations. Our approach is able to consistently filter GPT-3-generated explanations deemed acceptable by humans.

show abstract

Explanations for CommonsenseQA: New Dataset and Models

Cited by 26 publications

References 34 publications

Few-Shot Self-Rationalization with Natural Language Prompts

Few-Shot Self-Rationalization with Natural Language Prompts

A survey on improving NLP models with human explanations

Reframing Human-AI Collaboration for Generating Free-Text Explanations

Contact Info

Product

Resources

About