Topical language generation using transformers

Abstract

Large-scale transformer-based language models (LMs) demonstrate impressive capabilities in open-text generation. However, controlling the generated text’s properties such as the topic, style, and sentiment is challenging and often requires significant changes to the model architecture or retraining and fine-tuning the model on new supervised data. This paper presents a novel approach for topical language generation (TLG) by combining a pre-trained LM with topic modeling information. We cast the problem using Bayesian probability formulation with topic probabilities as a prior, LM probabilities as the likelihood, and TLG probability as the posterior. In learning the model, we derive the topic probability distribution from the user-provided document’s natural structure. Furthermore, we extend our model by introducing new parameters and functions to influence the quantity of the topical features presented in the generated text. This feature would allow us to easily control the topical properties of the generated text. Our experimental results demonstrate that our model outperforms the state-of-the-art results on coherency, diversity, and fluency while being faster in decoding.

A topic‐controllable keywords‐to‐text generator with knowledge base network

“…To fine‐tune our manually labelled training set, we utilise the topic model proposed by Zandie et al. [41], which achieves an accuracy of 0.87 on the test set. By using an automatic topic extractor during training, the target topic label s is calculated.…”

“…Our method of introducing topic labels involved manually annotating items with 100 categories, such as sports, beauty, and so on, using 3000 sentences. To fine-tune our manually labelled training set, we utilise the topic model proposed by Zandie et al [41], which achieves an accuracy of 0.87 on the test set. By using an automatic topic extractor during training, the target topic label s is calculated.…”

“…With a total parameter count of 72 MiB, our model parameters were randomly selected over a uniform distribution [0.1, 0.1]. We pre-train our model for 65 epochs with the Maximum Likelihood estimation model [41] and adversarial training [7] for 20 epochs. Our model is pre-trained with the Maximum Likelihood estimation model for 65 epochs and with adversarial training for 20 epochs.…”

Urdu paraphrase detection: A novel DNN-based implementation using a semi-automatically generated corpus

“…Automatic paraphrase detection is the task of deciding whether two given text fragments have the same meaning or not (Wang et al, 2021). Paraphrase detection has a number of applications, including question-answering (Noraset, Lowphansirikul, and Tuarob, 2021), natural language generation (Paris, Swartout, and Mann, 2013;Zandie and Mahoor, 2022), and intelligent tutoring systems (Forsythe, Bernard, and Goldsmith, 2006). In question-answering, multiple paraphrased answers could be considered as evidence for the correctness of an answer (Noraset et al, 2021).…”