Supervised Sequence Labelling with Recurrent Neural Networks

Abstract

Recurrent neural networks are powerful sequence learners. They are able to incorporate context information in a flexible way, and are robust to localised distortions of the input data. These properties make them well suited to sequence labelling, where input sequences are transcribed with streams of labels. Long short-term memory is an especially promising recurrent architecture, able to bridge long time delays between relevant input and output events, and thereby access long range context. The aim of this thesis is to advance the state-of-the-art in supervised sequence labelling with recurrent networks in general, and long short-term memory in particular. Its two main contributions are (1) a new type of output layer that allows recurrent networks to be trained directly for sequence labelling tasks where the alignment between the inputs and the labels is unknown, and (2) an extension of long short-term memory to multidimensional data, such as images and video sequences. Experimental results are presented on speech recognition, online and offline handwriting recognition, keyword spotting, image segmentation and image classification, demonstrating the advantages of advanced recurrent networks over other sequential algorithms, such as hidden Markov Models.ii

A Spatiotemporal Prediction Framework for Air Pollution Based on Deep RNN

“…Parameters of LSTM are trained using BPTT. Core structure of LSTM is illustrated as follows (Graves, 2012): Figure 3. Structure of LSTM memory block…”

Towards Automatic Generation of Product Reviews from Aspect-Sentiment Scores

“…RNN has been widely used for sequence generation tasks (Graves, 2012a;Schuster and Paliwal, 1997). RNN accepts sequence of inputs X = {x 1 , x 2 , x 3 , ..., x |X| }, and gets h t at time t according to Equation (2).…”

Analyzing the memory of BLSTM Neural Networks for enhanced emotion classification in dyadic spoken interactions

“…Since these gates allow for write, read, and reset operations within a memory block, an LSTM block can be interpreted as (differentiable) memory chip in a digital computer. The overall effect of the gate units is that the LSTM memory cells can store and access information over long periods of time and thus avoid the vanishing gradient problem (for details see [6]). …”

“…To extend the Jacobian to recurrent neural networks, we have to specify the timesteps (representing utterances) at which the input and output variables are measured. Thus, we calculate a four-dimensional matrix called the sequential Jacobian [6] to determine the sensitivity of the network outputs at time t to the inputs at time t :…”

“…Recently, bidirectional Long Short-Term Memory (BLSTM) neural networks were introduced in order to overcome the vanishing gradient problem of conventional Recurrent Neural Networks (RNNs) [5,6]. BLSTM neural networks make use of an arbitrary, self-learned amount of past and future contextual information.…”