A Generative Model for Joint Natural Language Understanding and Generation

Abstract

Natural language understanding (NLU) and natural language generation (NLG) are two fundamental and related tasks in building task-oriented dialogue systems with opposite objectives: NLU tackles the transformation from natural language to formal representations, whereas NLG does the reverse. A key to success in either task is parallel training data which is expensive to obtain at a large scale. In this work, we propose a generative model which couples NLU and NLG through a shared latent variable. This approach allows us to explore both spaces of natural language and formal representations, and facilitates information sharing through the latent space to eventually benefit NLU and NLG. Our model achieves state-of-the-art performance on two dialogue datasets with both flat and tree-structured formal representations. We also show that the model can be trained in a semi-supervised fashion by utilising unlabelled data to boost its performance.

Pretraining the Noisy Channel Model for Task-Oriented Dialogue

“…Task-Oriented Dialogue Models: Most taskoriented dialogue systems break down the task into three components: belief tracking (Henderson et al, 2013;Mrkšić et al, 2016;Rastogi et al, 2017;Nouri and Hosseini-Asl, 2018;Wu et al, 2019a;Zhou and Small, 2019;Heck et al, 2020), dialogue act prediction (Wen et al, 2017a;Tanaka et al, 2019), and response generation Budzianowski et al, 2018;Lippe et al, 2020). Traditionally, a modular approach is adopted, where these components are optimized independently (i.e., a pipeline design) or learned via multi-task learning (i.e., some parameters are shared among the components) (Wen et al, 2017b;Zhao et al, 2019;Mehri et al, 2019;Tseng et al, 2020;. However, it is known that improvements in one component do not necessarily lead to overall performance improvements (Ham et al, 2020), and the modular approach suffers from error propagation in practice (Liu and Lane, 2018).…”

Jointly Improving Language Understanding and Generation with Quality-Weighted Weak Supervision of Automatic Labeling

“…Learning with Semi-Supervision. Work on semi-supervised learning considers settings with some labeled data and a much larger set of unlabeled data, and then leverages both labeled the unlabeled data as in machine translation (Artetxe et al, 2017;Lample et al, 2017), data-to-text generation (Schmitt and Schütze, 2019;Qader et al, 2019) or more relevantly the joint learning framework for training NLU and NLG (Tseng et al, 2020;. Nonetheless, these approaches all assume that a large collection of text is available, which is an unrealistic assumption for the task due to the need for expert curation.…”

“…Natural language generation (NLG) is the task that transforms meaning representations (MR) into natural language descriptions (Reiter and Dale, 2000; Barzilay and Lapata, 2005); while natural language understanding (NLU) is the opposite process where text is converted into MR (Zhang and Wang, 2016). These two processes can thus constrain each other -recent exploration of the duality of neural natural language generation (NLG) and understanding (NLU) has led to successful semi-supervised learning techniques where both labeled and unlabeled data can be used for training Tseng et al, 2020;Schmitt and Schütze, 2019;Qader et al, 2019;.…”

“…On the other hand, learning with weak supervision from noisy labels offers a potential solution as it automatically builds imperfect training sets from low cost labeling rules or pretrained models (Zhou, 2018;Fries et al, 2020). Further, labeled data and large unlabeled data can be utilized in semi-supervised learning (Lample et al, 2017;Tseng et al, 2020), as a way to jointly improve both NLU and NLG models.…”

“…These noisy text labels are generated by a weak annotator, which is built upon recent works that directly fine-tune GPT-2 (Radford et al, 2019) on joint meaning representation (MR) and text (Mager et al, 2020;Harkous et al, 2020). Then, we jointly train the NLG and NLU models in a two-step process with semisupervised learning objectives (Tseng et al, 2020). First, we use pretrained models to estimate quality scores for each sample.…”