Semantic Noise Matters for Neural Natural Language Generation

Abstract

Neural natural language generation (NNLG) systems are known for their pathological outputs, i.e. generating text which is unrelated to the input specification. In this paper, we show the impact of semantic noise on state-of-theart NNLG models which implement different semantic control mechanisms. We find that cleaned data can improve semantic correctness by up to 97%, while maintaining fluency. We also find that the most common error is omitting information, rather than hallucination.

Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data

“…The former path is risky as it easily results in ungrammatical targets. The latter approach of enforcing a stronger alignment between inputs and outputs has been tried previously but it assumes a moderate amount of noise in the data (Nie et al, 2019;Dušek et al, 2019). Alternatively, one can leave the data as is and try to put more pressure on the decoder to pay attention to the input at every generation step (Tian et al, 2019).…”

How to Make Neural Natural Language Generation as Reliable as Templates in Task-Oriented Dialogue

“…But this is not a straightforward task. Specially designed regular expressions (Dušek et al, 2019a) or heuristics involving dependency relations (Oraby et al, 2019) must be used. Augmented Input Sequence Once the surface forms of each attribute-value pair in a target utterance are found, we add them to the input sequence, as shown in Figure 2.…”

“…To extract the surface form of each attribute-value pair from a target utterance, we used modified regular expressions from Dušek et al (2019a). The input sequence was constructed in the format of a single token representing an attribute-value pair followed by multiple tokens for the surface form, e.g.…”

“…To compare the diversity of the E2E training data with that of the generated text, we looked at the surface forms used to express each attribute-value pair. This is enabled by a set of regular expressions released by Dušek et al (2019a) 1 . The regular expressions capture the entire phrase used to express an attribute-value pair, focusing on the content words and attempting as much as possible to leave out the function words, e.g.…”

Structural Information Preserving for Graph-to-Text Generation

“…In particular, this can be very likely if "the girl wants" appears much more frequent than "the boy wants" in the training corpus. This is a very important issue, because of its wide existence across many neural graph-to-text arXiv:2102.06749v1 [cs.CL] 12 Feb 2021 generation models, hindering the usability of these models for real-world applications (Dušek et al, 2018(Dušek et al, , 2019Balakrishnan et al, 2019).…”