Can Neural Generators for Dialogue Learn Sentence Planning and Discourse Structuring?

Abstract

Responses in task-oriented dialogue systems often realize multiple propositions whose ultimate form depends on the use of sentence planning and discourse structuring operations. For example a recommendation may consist of an explicitly evaluative utterance e.g. Chanpen Thai is the best option, along with content related by the justification discourse relation, e.g. It has great food and service, that combines multiple propositions into a single phrase. While neural generation methods integrate sentence planning and surface realization in one endto-end learning framework, previous work has not shown that neural generators can:(1) perform common sentence planning and discourse structuring operations; (2) make decisions as to whether to realize content in a single sentence or over multiple sentences; (3) generalize sentence planning and discourse relation operations beyond what was seen in training. We systematically create large training corpora that exhibit particular sentence planning operations and then test neural models to see what they learn. We compare models without explicit latent variables for sentence planning with ones that provide explicit supervision during training. We show that only the models with additional supervision can reproduce sentence planning and discourse operations and generalize to situations unseen in training.

Evaluating the state-of-the-art of End-to-End Natural Language Generation: The E2E NLG challenge

“…Second, semantic errors were computed following Reed et al (2018), where we implemented a script to estimate the coverage automatically based on regular expression matching. 28 This allowed us to produce an independent estimate of the proportion of outputs with missing or added information (see Table 12).…”

“…28 This allowed us to produce an independent estimate of the proportion of outputs with missing or added information (see Table 12). Following Reed et al (2018), we also computed the slot error rate (SER) using this pattern-matching approach and the following formula: 29 SER = # missed + # added + # value errors + # repetitions # slots (5) Here, missed stands for slot values missing from the realisations, added denotes additional information not present in the MR (hallucinations), value errors denote correctly realised slots with incorrect values (e.g., specifying low price range instead of high),…”

“…While the absolute numbers for perfectly covered MRs are different from those estimated by humans, they mostly follow the same trend. The SER value is highly correlated with the proportion of perfectly covered MRs. 28 We based the patterns for the individual attribute-value pairs on Reed et al (2018)'s script and manually enhanced them using the first 500 instances of the E2E development set. 29 Note that the coverage and SER values produced by the script is only an estimate as the patterns for a given attribute-value pair will not cover all possible all correct ways to express it.…”

Constrained Decoding for Neural NLG from Compositional Representations in Task-Oriented Dialogue

“…This poses a dual challenge: First, since the MR does not specify these discourse relations, crowdworkers creating the dataset in turn have no instructions on when to use them, and must thus use their own judgment in creating a natural-sounding response. While the E2E organizers tout the resulting response variations as a plus, Reed et al (2018) find that current neural systems are unable to learn to express discourse relations effectively with this dataset, and explore ways of enriching input MRs to do so. Indeed, now that the E2E system outputs have been released, a search through outputs from all participating systems reveals only 43 outputs (0.4% out of 10080) containing contrastive tokens, on a test set containing about 300 contrastive samples.…”

Semantic Noise Matters for Neural Natural Language Generation

“…To produce a cleaned version of the E2E data, we used the original human textual references, but paired them with correctly matching MRs. 4 To this end, we reimplemented the slot matching script of Reed et al (2018), which tags MR slots and values using regular expressions. We tuned our expressions based on the first 500 instances from the E2E development set and ran the script on the full dataset, producing corrected MRs for all human references (see Figure 1).…”