Diversifying Neural Text Generation with Part-of-Speech Guided Softmax and Sampling

Abstract

Neural text generation models are likely to suffer from the low-diversity problem. Various decoding strategies and training-based methods have been proposed to promote diversity only by exploiting contextual features, but rarely do they consider incorporating syntactic structure clues. In this work, we propose using linguistic annotation, i.e., part-of-speech (POS), to guide the text generation. In detail, we introduce POS Guided Softmax (POSG-Softmax) to explicitly model two posterior probabilities: (i) next-POS, and (ii) nexttoken from the vocabulary of the target POS. A POS guided sampling strategy is further proposed to address the low-diversity problem by enriching the diversity of POS. Extensive experiments and human evaluations demonstrate that, compared with existing state-of-the-art methods, our proposed methods can generate more diverse text while maintaining comparable quality.

Syntax Controlled Knowledge Graph-to-Text Generation with Order and Semantic Consistency

“…image captioning and video captioning) to impose the syntactic constraint. In the neural text generation work (Yang and Wan, 2021), the authors propose to use POS guided softmax function as the linguistic prior information for modeling the posterior probabilities of next-POS and next-token, in order to increase text generation diversity. In the image caption, Bugliarello et al (Bugliarello and Elliott, 2021) claim that incorporating POS tag information in the sentence generation process consistently improves the quality of the generated text.…”