Dynamic Topic Tracker for KB-to-Text Generation

Fu, Zihao; Lam, Wai; Jameel, Shoaib

doi:10.18653/v1/2020.coling-main.215

“…However, these models focus on aligning source words to target words, and no existing models have been proposed to directly calculate supportiveness for generation tasks. In generation systems, Fu et al (2020b) propose to dynamically align the current generation state with topics to improve the generation performance. However, it still can not directly align to the input source words.…”

Section: Related Workmentioning

confidence: 99%

Partially-Aligned Data-to-Text Generation with Distant Supervision

Fu

¹

,

Shi²,

Lam

³

et al. 2020

Preprint

Self Cite

2

0

View full text Add to dashboard Cite

The Data-to-Text task aims to generate humanreadable text for describing some given structured data enabling more interpretability. However, the typical generation task is confined to a few particular domains since it requires wellaligned data which is difficult and expensive to obtain. Using partially-aligned data is an alternative way of solving the dataset scarcity problem. This kind of data is much easier to obtain since it can be produced automatically. However, using this kind of data induces the over-generation problem posing difficulties for existing models, which tends to add unrelated excerpts during the generation procedure. In order to effectively utilize automatically annotated partially-aligned datasets, we extend the traditional generation task to a refined task called Partially-Aligned Data-to-Text Generation (PADTG) which is more practical since it utilizes automatically annotated data for training and thus considerably expands the application domains. To tackle this new task, we propose a novel distant supervision generation framework. It firstly estimates the input data's supportiveness for each target word with an estimator and then applies a supportiveness adaptor and a rebalanced beam search to harness the over-generation problem in the training and generation phases respectively. We also contribute a partially-aligned dataset 1 by sampling sentences from Wikipedia and automatically extracting corresponding KB triples for each sentence from Wikidata. The experimental results show that our framework outperforms all baseline models as well as verify the feasibility of utilizing partially-aligned data.

show abstract

A Theoretical Analysis of the Repetition Problem in Text Generation

Fu

¹

,

Lam

²

,

So

³

et al. 2021

AAAI

View full text Add to dashboard Cite

Text generation tasks, including translation, summarization, language models, and etc. see rapid growth during recent years. Despite the remarkable achievements, the repetition problem has been observed in nearly all text generation models undermining the generation performance extensively. To solve the repetition problem, many methods have been proposed, but there is no existing theoretical analysis to show why this problem happens and how it is resolved. In this paper, we propose a new framework for theoretical analysis for the repetition problem. We first define the Average Repetition Probability (ARP) to characterize the repetition problem quantitatively. Then, we conduct an extensive analysis of the Markov generation model and derive several upper bounds of the average repetition probability with intuitive understanding. We show that most of the existing methods are essentially minimizing the upper bounds explicitly or implicitly. Grounded on our theory, we show that the repetition problem is, unfortunately, caused by the traits of our language itself. One major reason is attributed to the fact that there exist too many words predicting the same word as the subsequent word with high probability. Consequently, it is easy to go back to that word and form repetitions and we dub it as the high inflow problem. Furthermore, we extend our analysis to broader generation models by deriving a concentration bound of the average repetition probability for a general generation model. Finally, based on the theoretical upper bounds, we propose a novel rebalanced encoding approach to alleviate the high inflow problem and thus reducing the upper bound. The experimental results show that our theoretical framework is applicable in general generation models and our proposed rebalanced encoding approach alleviates the repetition problem significantly in both the translation task and the language modeling task. The source code of this paper can be obtained from https://github.com/fuzihaofzh/repetition-problem-nlg.

show abstract

Partially-Aligned Data-to-Text Generation with Distant Supervision

Fu

¹

,

Shi

²

,

Lam

³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

The Data-to-Text task aims to generate humanreadable text for describing some given structured data enabling more interpretability. However, the typical generation task is confined to a few particular domains since it requires wellaligned data which is difficult and expensive to obtain. Using partially-aligned data is an alternative way of solving the dataset scarcity problem. This kind of data is much easier to obtain since it can be produced automatically. However, using this kind of data induces the over-generation problem posing difficulties for existing models, which tends to add unrelated excerpts during the generation procedure. In order to effectively utilize automatically annotated partially-aligned datasets, we extend the traditional generation task to a refined task called Partially-Aligned Data-to-Text Generation (PADTG) which is more practical since it utilizes automatically annotated data for training and thus considerably expands the application domains. To tackle this new task, we propose a novel distant supervision generation framework. It firstly estimates the input data's supportiveness for each target word with an estimator and then applies a supportiveness adaptor and a rebalanced beam search to harness the over-generation problem in the training and generation phases respectively. We also contribute a partially-aligned dataset 1 by sampling sentences from Wikipedia and automatically extracting corresponding KB triples for each sentence from Wikidata. The experimental results show that our framework outperforms all baseline models as well as verify the feasibility of utilizing partially-aligned data. 1 The data and source code of this paper can be obtained from https://github.com/fuzihaofzh/ distant_supervision_nlg Company of Heroes is a strategy video game developed in Canada. Company of Heroes is a strategy video game developed in Canada. Company of Heroes, genre, strategy video game Company of Heroes is a strategy video game developed in Canada.

show abstract

Dynamic Topic Tracker for KB-to-Text Generation

Cited by 3 publications

References 75 publications

Partially-Aligned Data-to-Text Generation with Distant Supervision

Partially-Aligned Data-to-Text Generation with Distant Supervision

A Theoretical Analysis of the Repetition Problem in Text Generation

Partially-Aligned Data-to-Text Generation with Distant Supervision

Contact Info

Product

Resources

About