Few-Shot Self-Rationalization with Natural Language Prompts

Marasović, Ana; Beltagy, Iz; Downey, Doug; Peters, Matthew E.

doi:10.18653/v1/2022.findings-naacl.31

Cited by 19 publications

(26 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…UnifiedQA-v2-11B is also better than FLAN-T5-XXL (a 11Bparameter model as well). Given that UnifiedQA-v1 (Khashabi et al, 2020) has been effective for tasks beyond QA (Bragg et al, 2021;Marasović et al, 2022) models are strong but overlooked baselines in recent works on large-scale models.…”

Section: Resultsmentioning

confidence: 99%

CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation

Ravichander¹,

Gardner²,

Marasović³

2022

Preprint

View full text Add to dashboard Cite

The full power of human language-based communication cannot be realized without negation. All human languages have some form of negation. Despite this, negation remains a challenging phenomenon for current natural language understanding systems. To facilitate the future development of models that can process negation effectively, we present CONDAQA, the first English reading comprehension dataset which requires reasoning about the implications of negated statements in paragraphs. We collect paragraphs with diverse negation cues, then have crowdworkers ask questions about the implications of the negated statement in the passage. We also have workers make three kinds of edits to the passage-paraphrasing the negated statement, changing the scope of the negation, and reversing the negation-resulting in clusters of question-answer pairs that are difficult for models to answer with spurious shortcuts. CONDAQA features 14,182 question-answer pairs with over 200 unique negation cues and is challenging for current state-of-the-art models. The best performing model on CONDAQA (UNIFIEDQA-V2-3B) achieves only 42% on our consistency metric, well below human performance which is 81%. We release our dataset, along with fully-finetuned, few-shot, and zero-shot evaluations, to facilitate the development of future NLP methods that work on negated language. * Work undertaken while Abhilasha Ravichander and Ana Marasović were at the Allen Institute for AI.1 COntrastively-annotated Negation DAtaset of Question-Answer pairs

show abstract

Section: Resultsmentioning

confidence: 99%

CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation

Ravichander¹,

Gardner²,

Marasović³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Large language models like GPT-3 [7] have popularized few-shot prompting in natural language processing (NLP), where several inputoutput pairs are used as context for the language model to understand the task and generate predictions for a new example. Some prompting techniques [16,47,60,73] have also been developed for more effective or transparent reasoning in NLP. Later, prompting was brought to the vision community [24,31,34,59,74,75].…”

Section: Related Workmentioning

confidence: 99%

See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning

Zhenfang¹,

Qinhong²,

Shen³

et al. 2023

Preprint

View full text Add to dashboard Cite

Large pre-trained vision and language models have demonstrated remarkable capacities for various tasks. However, solving the knowledge-based visual reasoning tasks remains challenging, which requires a model to comprehensively understand image content, connect external world knowledge, and perform step-by-step reasoning to answer the questions correctly. To this end, we propose a novel framework named Interactive Prompting Visual Reasoner (IPVR) for few-shot knowledge-based visual reasoning. IPVR contains three stages, see, think and confirm. The see stage scans the image and grounds the visual concept candidates with a visual perception model. The think stage adopts a pre-trained large language model (LLM) to attend to the key concepts from candidates adaptively. It then transforms them into text context for prompting with a visual captioning model and adopts the LLM to generate the answer. The confirm stage further uses the LLM to generate the supporting rationale to the answer, verify the generated rationale with a cross-modality classifier and ensure that the rationale can infer the predicted output consistently. We conduct experiments on a range of knowledge-based visual reasoning datasets. We found our IPVR enjoys several benefits, 1). it achieves better performance than the previous few-shot learning baselines; 2). it enjoys the total transparency and trustworthiness of the whole reasoning process by providing rationales for each reasoning step; 3). it is computation-efficient compared with other fine-tuning baselines.

show abstract

“…Learning From In-Context Instructions. The few-shot performance of LMs can be enhanced by learning from in-context instructions (Sanh et al, 2021;Liu et al, 2021b), in the forms of task descriptions (Mishra et al, 2021;Raffel et al, 2019), answer demonstrations (Brown et al, 2020), targeting formats (Marasović et al, 2021), etc., which can be positioned before or even after (Lampinen et al, 2022) the answer. Recent studies have shown improved results by including decomposed reasoning steps into the instructions (Nye et al, 2021;Narang et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits

Liu¹,

Jia²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

We present SECOND THOUGHTS, a new learning paradigm that enables language models (LMs) to re-align with human values. By modeling the chain-of-edits between value-unaligned and value-aligned text, with LM fine-tuning and additional refinement through reinforcement learning, SECOND THOUGHTS not only achieves superior performance in three value alignment benchmark datasets but also shows strong human-value transfer learning ability in few-shot scenarios. The generated editing steps also offer better interpretability and ease for interactive error correction. Extensive human evaluations further confirm its effectiveness. * Work done during the internship at Dartmouth College. 2 We take the log-probability predicted by the LM, log Pr(y|x), which is the conditional log-probability of generating option y given input context x. We then compute its exponential for better readability. Such a protocol is also adopted by BIG-Bench: https://github.com/google/BIG-bench.36th Conference on Neural Information Processing Systems (NeurIPS 2022).the source to produce the target (Figure 2 (b)). This way the model learns how to recover from a value-unaligned, poisoned context during the generation phase. Augmented Edits ModelingDP-based Edits Inference. Given two text strings, source and target, one can find unlimited ways to edit source to produce target. Thus, we apply two constraints onto the editing: (1) the edits should be combinations of generic editing operations-inserting, deleting, and replacing a single token; (2) each edit operation has a cost and our goal is to infer the chain-of-edits that has minimum cost. Under these constraints, the edits inference problem can be converted to a token-level "edit distance problem" (Jurafsky, 2000), which can be solved by dynamic programming (DP). We modify the algorithm to be able to receive customized editing costs (e.g., insert-1, delete-1, replace-2), to try to model different preferences on editing. We use special tokens to mark the start/end of editing and the new content to be inserted/replaced, and develop a decipher module that can translate the edit operations produced by DP into natural language (see §A.1 for a visualization of the whole process, and §A.3 for more discussion on edit based models).Augmented Edits Modeling (AEM). To augment the edits, we run the DP algorithm on the same source and target pairs with a variety of editing costs 4 to create a collection of chain-of-edits for each source-target pair, which we call positive demonstrations (y + ). We then fine-tune an LM on these source-edits-target text inputs (recall that the edits are turned into natural language). We call this Augmented Edits Modeling (AEM). Different from common language modeling, AEM includes the labor-free decomposition (i.e., the editing steps) into the training object, whereas prior works either train on costly manually-created decomposition (Ouyang et al., 2022; or, rather than training, prompt with such decomposition Nye et al., 2021). We also construct negative demonstrations (y − ) by us...

show abstract

Few-Shot Self-Rationalization with Natural Language Prompts

Cited by 19 publications

References 42 publications

CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation

CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation

See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning

Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits

Contact Info

Product

Resources

About