A Latent Variable Recurrent Neural Network for Discourse-Driven Language Models

Abstract

This paper presents a novel latent variable recurrent neural network architecture for jointly modeling sequences of words and (possibly latent) discourse relations between adjacent sentences. A recurrent neural network generates individual words, thus reaping the benefits of discriminatively-trained vector representations. The discourse relations are represented with a latent variable, which can be predicted or marginalized, depending on the task. The resulting model can therefore employ a training objective that includes not only discourse relation classification, but also word prediction. As a result, it outperforms state-ofthe-art alternatives for two tasks: implicit discourse relation classification in the Penn Discourse Treebank, and dialog act classification in the Switchboard corpus. Furthermore, by marginalizing over latent discourse relations at test time, we obtain a discourse informed language model, which improves over a strong LSTM baseline.

The Context-Dependent Additive Recurrent Neural Net

“…The Penn Tree Bank corpus provides discourse relation annotation between the spans of text. We used the preprocessed data by Ji et al (2016b), where the explicit discourse relations are mapped into a dummy relation. Our data splits are the same as those described in the baselines (Ji et al, 2016a,b).…”

“…We compare our system with the Recurrent Neural Net (RNNLM) with LSTM unit (Ji et al, 2016a), the Document Contextual Lan- guage Model (DCLM) (Ji et al, 2016a) and the Discourse Relation Language Model (DRLM) (Ji et al, 2016b). The RNNLM's architecture is the same as that described in (Mikolov et al, 2013) with sigmoid non-linearity replaced by LSTM.…”

A Generative Attentional Neural Network Model for Dialogue Act Classification

“…Some works in DA classification treat each utterance as an independent instance (Julia et al, 2010;Gambäck et al, 2011), which leads to ignoring important long-range dependencies in the dialogue history. Other works have captured inter-utterance relationships using models such as Hidden Markov Models (HMMs) (Stolcke et al, 2000;Surendran and Levow, 2006) or Recurrent Neural Networks (RNNs) (Kalchbrenner and Blunsom, 2013;Ji et al, 2016), where RNNs have been particularly successful.…”

Preserving Distributional Information in Dialogue Act Classification

“…There have been many works on DA classification applied to these two datasets; some focus on textual data (Kalchbrenner and Blunsom, 2013;Stolcke et al, 2000), while others explore speech data (Julia et al, 2010). The classification methods used can be broadly divided into instancebased methods (Julia et al, 2010;Gambäck et al, 2011) and sequence-labeling methods (Stolcke et al, 2000;Kalchbrenner and Blunsom, 2013;Ji et al, 2016;Shen and Lee, 2016;Tran et al, 2017). Instance-based methods treat each utterance as an independent data point, which allows the application of general machine learning models, such as Support Vector Machines.…”

“…Instance-based methods treat each utterance as an independent data point, which allows the application of general machine learning models, such as Support Vector Machines. Sequencelabeling methods include methods based on Hidden Markov Models (HMMs) (Stolcke et al, 2000) and neural networks (Kalchbrenner and Blunsom, 2013;Ji et al, 2016;Shen and Lee, 2016;Tran et al, 2017).…”