Prediction of LSTM-RNN Full Context States as a Subtask for N-Gram Feedforward Language Models

Abstract

Long short-term memory (LSTM) recurrent neural network language models compress the full context of variable lengths into a fixed size vector. In this work, we investigate the task of predicting the LSTM hidden representation of the full context from a truncated n-gram context as a subtask for training an n-gram feedforward language model. Since this approach is a form of knowledge distillation, we compare two methods. First, we investigate the standard transfer based on the Kullback-Leibler divergence of the output distribution of the feedforward model from that of the LSTM. Second, we minimize the mean squared error between the hidden state of the LSTM and that of the n-gram feedforward model. We carry out experiments on different subsets of the Switchboard speech recognition dataset for feedforward models with a short (5-gram) and a medium (10-gram) context length. We show that we get improvements in perplexity and word error rate of up to 8% and 4% relative for the medium model, while the improvements are only marginal for the short model.

Language Modeling with Deep Transformers

“…In addition, we examine the type of activation function ( Table 4). As opposed to previous work on feed-forward language models using GLUs [28,34], we do not observe faster convergence. As we observe that the impact of choice of activation functions on the perplexity is overall limited, all our other models use the standard ReLU.…”

Interval Prediction of the Permeability of Granite Bodies in a High-Level Radioactive Waste Disposal Site Using LSTM-RNNs and Probability Distribution

“…When the information flow enters the algorithm network, the redundant information that does not meet the algorithm rules is placed in the forget gate and removed (Li et al, 2021c). In practice, the LSTM-RNN algorithm can be expressed by Eqs 9-13 (Irie et al, 2018):…”