Pretrained Language Model for Text Generation: A Survey

Li, Junyi; Tang, Tianyi; Zhao, Wayne Xin; Wen, Ji-Rong

doi:10.24963/ijcai.2021/612

Cited by 94 publications

(58 citation statements)

References 5 publications

(12 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this work, we use pre-trained natural language models in order to extract the contextual semantic patterns in a collection of open-responses. Pre-trained natural language models [10] are commonly used for a wide range of tasks such as text generation [11], building dialogue systems [12], text classification [13], hate speech detection [14], sentiment analysis [15], named entity recognition [16], question answering [17], and text summarization [18,19].…”

Section: Methodsmentioning

confidence: 99%

Providing Insights for Open-Response Surveys via End-to-End Context-Aware Clustering

Esmaeilzadeh¹,

Williams²,

Shamsi³

et al. 2022

Preprint

View full text Add to dashboard Cite

Teachers often conduct surveys in order to collect data from a predefined group of students to gain insights into topics of interest. When analyzing surveys with open-ended textual responses, it is extremely time-consuming, labor-intensive, and difficult to manually process all the responses into an insightful and comprehensive report. In the analysis step, traditionally, the teacher has to read each of the responses and decide on how to group them in order to extract insightful information. Even though it is possible to group the responses only using certain keywords, such an approach would be limited since it not only fails to account for embedded contexts but also cannot detect polysemous words or phrases and semantics that are not expressible in single words. In this work, we present a novel end-to-end context-aware framework that extracts, aggregates, and abbreviates embedded semantic patterns in open-response survey data. Our framework relies on a pre-trained natural language model in order to encode the textual data into semantic vectors. The encoded vectors then get clustered either into an optimally tuned number of groups or into a set of groups with pre-specified titles. In the former case, the clusters are then further analyzed to extract a representative set of keywords or summary sentences that serve as the labels of the clusters. In our framework, for the designated clusters, we finally provide context-aware wordclouds that demonstrate the semantically prominent keywords within each group. Honoring user privacy, we have successfully built the on-device implementation of our framework suitable for real-time analysis on mobile devices and have tested it on a synthetic dataset. Our framework reduces the costs at-scale by automating the process of extracting the most insightful information pieces from survey data.

show abstract

Section: Methodsmentioning

confidence: 99%

Providing Insights for Open-Response Surveys via End-to-End Context-Aware Clustering

Esmaeilzadeh¹,

Williams²,

Shamsi³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Our solution is inspired by the excellent few-shot capabilities of pretrained language models (PLMs) on language understanding and generation tasks (Brown et al, 2020;Chen et al, 2020;Li et al, 2021a). Pretrained on the large-scale corpora, PLMs encode vast amounts of world knowledge into their parameters (Li et al, 2021b), which is potentially beneficial to understand and describe the KG facts in our task.…”

Section: Kg Descriptive Textmentioning

confidence: 99%

“…Recent years have witnessed prominent achievement of PLMs in NLP tasks (Devlin et al, 2019;Radford et al, 2019). Pretrained on massive corpora, pretrained models showcase unprecedented generalization ability to solve related downstream tasks (Li et al, 2021b). However, most of existing PLMs were conditioned on text data (Radford et al, 2019;Lewis et al, 2020), lacking consideration of structured data input.…”

Section: Related Workmentioning

confidence: 99%

Few-shot Knowledge Graph-to-Text Generation with Pretrained Language Models

Tang

Zhao

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

This paper studies how to automatically generate a natural language text that describes the facts in knowledge graph (KG). Considering the few-shot setting, we leverage the excellent capacities of pretrained language models (PLMs) in language understanding and generation. We make three major technical contributions, namely representation alignment for bridging the semantic gap between KG encodings and PLMs, relation-biased KG linearization for deriving better input representations, and multi-task learning for learning the correspondence between KG and text. Extensive experiments on three benchmark datasets have demonstrated the effectiveness of our model on KG-to-text generation task. In particular, our model outperforms all comparison methods on both fully-supervised and fewshot settings. Our code and datasets are available at https://github.com/RUCAIBox/ Few-Shot-KG2Text.

show abstract

“…Pre-trained language models (PLMs) (Peters et al, 2018;Devlin et al, 2019), are now used in almost all NLP applications, e.g., machine translation (Li et al, 2021), question answering (Zhang et al, 2020), dialogue systems (Ni et al, 2021), and sentiment analysis (Minaee et al, 2020). They have sometimes been referred to as "foundation models" (Bommasani et al, 2021) due to their significant impact on research and industry.…”

Section: Introductionmentioning

confidence: 99%

Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages

Aghazadeh¹,

Fayyaz²,

Yaghoobzadeh³

2022

Preprint

View full text Add to dashboard Cite

Human languages are full of metaphorical expressions. Metaphors help people understand the world by connecting new concepts and domains to more familiar ones. Large pretrained language models (PLMs) are therefore assumed to encode metaphorical knowledge useful for NLP systems. In this paper, we investigate this hypothesis for PLMs, by probing metaphoricity information in their encodings, and by measuring the cross-lingual and crossdataset generalization of this information. We present studies in multiple metaphor detection datasets and in four languages (i.e., English, Spanish, Russian, and Farsi). Our extensive experiments suggest that contextual representations in PLMs do encode metaphorical knowledge, and mostly in their middle layers. The knowledge is transferable between languages and datasets, especially when the annotation is consistent across training and testing sets. Our findings give helpful insights for both cognitive and NLP scientists.

show abstract

Pretrained Language Model for Text Generation: A Survey

Cited by 94 publications

References 5 publications

Providing Insights for Open-Response Surveys via End-to-End Context-Aware Clustering

Providing Insights for Open-Response Surveys via End-to-End Context-Aware Clustering

Few-shot Knowledge Graph-to-Text Generation with Pretrained Language Models

Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages

Contact Info

Product

Resources

About