Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning

Qin, Lianhui; Shwartz, Vered; West, Peter; Bhagavatula, Chandra; Hwang, Jena D.; Bras, Ronan Le; Bosselut, Antoine; Choi, Yejin

doi:10.18653/v1/2020.emnlp-main.58

Cited by 39 publications

(63 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This work utilizes the technology from causality (Pearl, 2009) which is a powerful to study the cause-effect between variables and results (Pearl, 2009). Supported by this technology, recent studies recognize the spurious correlations in neural models (Feng et al, 2018;Gururangan et al, 2018), and several works introduce causal mechanism into computer vision (Tang et al, 2020;Qi et al, 2020) and natural language processing (Zeng et al, 2020;Wu et al, 2020;Qin et al, 2020;Fu et al, 2020). As far as we know, we are the first work to analyze the instability in OpenRE from the perspective of causality.…”

Section: Related Workmentioning

confidence: 99%

Element Intervention for Open Relation Extraction

Liu¹,

Yan²,

Lin³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Open relation extraction aims to cluster relation instances referring to the same underlying relation, which is a critical step for general relation extraction. Current OpenRE models are commonly trained on the datasets generated from distant supervision, which often results in instability and makes the model easily collapsed. In this paper, we revisit the procedure of OpenRE from a causal view. By formulating OpenRE using a structural causal model, we identify that the above-mentioned problems stem from the spurious correlations from entities and context to the relation type.To address this issue, we conduct Element Intervention, which intervene on the context and entities respectively to obtain the underlying causal effects of them. We also provide two specific implementations of the interventions based on entity ranking and context contrasting. Experimental results on unsupervised relation extraction datasets show our method to outperform previous state-of-the-art methods and is robust across different datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

Element Intervention for Open Relation Extraction

Liu¹,

Yan²,

Lin³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

show abstract

“…Non-monotonic generation and refinement. Another relevant line of research is non-monotonic generation Gu et al, 2019;, infilling (Zhu et al, 2019;Shen et al, 2020;Qin et al, 2020), or refinement (Lee et al, 2018;Novak et al, 2016;Mansimov et al, 2019;Kasai et al, 2020) that differs from the restricted left-toright generation in conventional LMs. Again, those approaches largely depend on specialized architectures and inference, making them difficult to be integrated with the powerful pretrained LMs.…”

Section: Related Workmentioning

confidence: 99%

Progressive Generation of Long Text with Pretrained Language Models

Tan¹,

Yang²,

Al-Shedivat³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Large-scale language models (LMs) pretrained on massive corpora of text, such as GPT-2, are powerful open-domain text generators. However, as our systematic examination reveals, it is still challenging for such models to generate coherent long passages of text (e.g., 1000 tokens), especially when the models are fine-tuned to the target domain on a small corpus. Previous planning-then-generation methods also fall short of producing such long text in various domains. To overcome the limitations, we propose a simple but effective method of generating text in a progressive manner, inspired by generating images from low to high resolution. Our method first produces domain-specific content keywords and then progressively refines them into complete passages in multiple stages. The simple design allows our approach to take advantage of pretrained LMs at each stage and effectively adapt to any target domain given only a small set of examples. We conduct a comprehensive empirical study with a broad set of evaluation metrics, and show that our approach significantly improves upon the fine-tuned large LMs and various planning-then-generation methods in terms of quality and sample efficiency. Human evaluation also validates that our model generations are more coherent. 1

show abstract

“…This naturally fits RE-FLECTIVE DECODING, which fills in contextual gaps. Recent work has directly addressed this task (Qin et al, 2020) while the infilling literature is also quite applicable (Donahue et al, 2020). We compare to both of these methods on abductive infilling, showing superior results.…”

Section: Novelty In Paraphrasingmentioning

confidence: 99%

Reflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Models

West

Holtzman

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

Self Cite

View full text Add to dashboard Cite

Publicly available, large pretrained Language Models (LMs) generate text with remarkable quality, but only sequentially from left to right. As a result, they are not immediately applicable to generation tasks that break the unidirectional assumption, such as paraphrasing or text-infilling, necessitating task-specific supervision.In this paper, we present REFLECTIVE DE-CODING, a novel unsupervised algorithm that allows for direct application of unidirectional LMs to non-sequential tasks. Our 2-step approach requires no supervision or even parallel corpora, only two off-the-shelf pretrained LMs in opposite directions: forward and backward. First, in the contextualization step, we use LMs to generate ensembles of past and future contexts which collectively capture the input (e.g. the source sentence for paraphrasing). Second, in the reflection step, we condition on these "context ensembles", generating outputs that are compatible with them. Comprehensive empirical results 1 demonstrate that REFLECTIVE DECODING outperforms strong unsupervised baselines on both paraphrasing and abductive text infilling, significantly narrowing the gap between unsupervised and supervised methods. REFLECTIVE DECODING surpasses multiple supervised baselines on various metrics including human evaluation.

show abstract

Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning

Cited by 39 publications

References 46 publications

Element Intervention for Open Relation Extraction

Element Intervention for Open Relation Extraction

Progressive Generation of Long Text with Pretrained Language Models

Reflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Models

Contact Info

Product

Resources

About