Recurrent Neural Networks (RNNs): Architectures, Training Tricks, and Introduction to Influential Research

Abstract

Recurrent neural networks (RNNs) are neural network architectures with hidden state and which use feedback loops to process a sequence of data that ultimately informs the final output. Therefore, RNN models can recognize sequential characteristics in the data and help to predict the next likely data point in the data sequence. Leveraging the power of sequential data processing, RNN use cases tend to be connected to either language models or time-series data analysis. However, multiple popular RNN architectures have been introduced in the field, starting from SimpleRNN and LSTM to deep RNN, and applied in different experimental settings. In this chapter, we will present six distinct RNN architectures and will highlight the pros and cons of each model. Afterward, we will discuss real-life tips and tricks for training the RNN models. Finally, we will present four popular language modeling applications of the RNN models –text classification, summarization, machine translation, and image-to-text translation– thereby demonstrating influential research in the field.

Exploring Types of Photonic Neural Networks for Imaging and Computing—A Review

“…RNNs characterize a specialized class of ANNs engineered to manage sequential data [ 49 , 50 ]. They achieve this by incorporating connections that create directed cycles within the network graph, facilitating dynamic temporal behavior and the processing of sequences of variable lengths.…”

A Systematic Review on Semantic Role Labeling for Information Extraction in Low-Resource Data

“…Traditional RNNs suffer from the vanishing gradient problem, which makes it difficult for the model to capture long-range dependencies in a sequence. LSTM is an advancement over traditional RNNs, designed to overcome the vanishing gradient problem by introducing memory cells, input gates, forget gates and output gates to control the flow of information into and out of the cells [91]. The RNN model can transfer information more efficiently by using skip connections.…”

“…The reset gate regulates the extent to which past information must be forgotten to update the relevant memory, while the update gate regulates the retention or abandonment of information and the integration of new information from input into existing memory. With these two gates, GRU provides adaptive control over the flow of information in recurrent networks [91]. Marzinotto et al [56] use four layers of bidirectional GRU and are equipped with various more complex features for SRL tasks, including morphology, surface characteristics, and syntax.…”

Comparison of simulation and predictive efficacy for hemorrhagic fever with renal syndrome incidence in mainland China based on five time series models

“…These crucial state transitions are meticulously orchestrated by a set of adaptive gating units, which include the input, forget, and output gates, each performing a specific regulatory function to ensure the fidelity of information flow across the temporal expanse of the sequence. There are three kinds of gates in the LSTM layer, input gate, forget gate, and output gate (35). Figure 1 illustrates the flow of data at time step t and shows how the gates forget, update, and output the cell and hidden states.…”