INSET: Sentence Infilling with INter-SEntential Transformer

Huang, Yichen; Zhang, Yizhe

doi:10.18653/v1/2020.acl-main.226

Cited by 26 publications

(23 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…C ⊥ ⊥ T: Story generation (Fan et al, 2019), text infilling (Fedus et al, 2018;Huang et al, 2019), paragraph bridging (Kang et al, 2019), and our proposed PARCOM are very challenging tasks where context and target have no overlap (open-ended), but they should be coherently connected. Keskar et al (2019) conditioned language models with topical words to control the target text.…”

Section: Related Workmentioning

confidence: 99%

“…However, they are given the topical content as input (content guidance), while our SSPlanner directly predicts plan words from context (content prediction). Fedus et al (2018); Huang et al (2019) developed various methods for text infilling task. Very similar to our task, Kang et al (2019) developed language models informed by discourse relations on the bridging task; given the first and last sentences, predicting the intermediate sentences (bidirectional flow).…”

Section: Related Workmentioning

confidence: 99%

“…However, they did not explicitly predict content words given context nor use them as a self-supervision signal in training. Unlike random masking in Keskar et al (2019); Huang et al (2019), we propose a better data augmentation training method via permutation masking.…”

Section: Related Workmentioning

confidence: 99%

“…Prior tasks fall into classification or ranking problems, such as narrative close task (Chambers and Jurafsky, 2008;Mostafazadeh et al, 2016), sentence ordering (Barzilay and Lapata, 2008), and next sentence prediction (Devlin et al, 2019). Some recent works focused on designing generation tasks: story generation (Fan et al, 2019), text infilling (Huang et al, 2019;Fedus et al, 2018;Hua and Wang, 2019), or paragraph bridging (Kang et al, 2019). However, most of them suffer from predicting appropriate topical content given limited context, due to the limited usage of self-supervision signals from the paragraph.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Plan ahead: Self-Supervised Text Planning for Paragraph Completion Task

Kang

Hovy

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Despite the recent success of contextualized language models on various NLP tasks, language model itself cannot capture textual coherence of a long, multi-sentence document (e.g., a paragraph). Humans often make structural decisions on what and how to say about before making utterances. Guiding surface realization with such high-level decisions and structuring text in a coherent way is essentially called a planning process. Where can the model learn such high-level coherence? A paragraph itself contains various forms of inductive coherence signals called self-supervision in this work, such as sentence orders, topical keywords, rhetorical structures, and so on. Motivated by that, this work proposes a new paragraph completion task PAR-COM; predicting masked sentences in a paragraph. However, the task suffers from predicting and selecting appropriate topical content with respect to the given context. To address that, we propose a self-supervised text planner SSPlanner that predicts what to say first (content prediction), then guides the pretrained language model (surface realization) using the predicted content. SSPlanner outperforms the baseline generation models on the paragraph completion task in both automatic and human evaluation. We also find that a combination of noun and verb types of keywords is the most effective for content selection. As more number of content keywords are provided, overall generation quality also increases.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Plan ahead: Self-Supervised Text Planning for Paragraph Completion Task

Kang

Hovy

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

show abstract

“…For example, inspired by skipgrams (Mikolov et al, 2013), proposed to train a sequence-to-sequence model to generate sentences before and after a sentence, and use the encoder to compute sentence representations. Inspired by masked language modeling in BERT, Zhang et al (2019) and Huang et al (2020) presented methods to learn contextualized sentence representations through the task of restoring a masked sentence from its context.…”

Section: Introductionmentioning

confidence: 99%

Contextualized and Generalized Sentence Representations by Contrastive Self-Supervised Learning: A Case Study on Discourse Relation Analysis

Kiyomaru¹,

Kurohashi²

2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

We propose a method to learn contextualized and generalized sentence representations using contrastive self-supervised learning. In the proposed method, a model is given a text consisting of multiple sentences. One sentence is randomly selected as a target sentence. The model is trained to maximize the similarity between the representation of the target sentence with its context and that of the masked target sentence with the same context. Simultaneously, the model minimize the similarity between the latter representation and the representation of a random sentence with the same context. We apply our method to discourse relation analysis in English and Japanese and show that it outperforms strong baseline methods based on BERT, XLNet, and RoBERTa.

show abstract