Residual Recurrent Neural Networks for Learning Sequential Representations

Abstract

Recurrent neural networks (RNN) are efficient in modeling sequences for generation and classification, but their training is obstructed by the vanishing and exploding gradient issues. In this paper, we reformulate the RNN unit to learn the residual functions with reference to the hidden state instead of conventional gated mechanisms such as long short-term memory (LSTM) and the gated recurrent unit (GRU). The residual structure has two main highlights: firstly, it solves the gradient vanishing and exploding issues for large time-distributed scales; secondly, the residual structure promotes the optimizations for backward updates. In the experiments, we apply language modeling, emotion classification and polyphonic modeling to evaluate our layer compared with LSTM and GRU layers. The results show that our layer gives state-of-the-art performance, outperforms LSTM and GRU layers in terms of speed, and supports an accuracy competitive with that of the other methods.

Designing, Developing and Validating a Forecasting Method for the Month Ahead Hourly Electricity Consumption in the Case of Medium Industrial Consumers

“…In the case when this process takes place, due to the fact that many derivatives having low-values are multiplied when computing the chain rule, the gradient vanishes to 0. In the case of the LSTM ANNs, the derivative of the identical function from Equation 2is the constant function 1 and this fact represents a certain advantage in the case when the LSTM is trained based on the backpropagation, because in this case the gradient is not vanishing [47].…”

Glomerular Filtration Rate Estimation by a Novel Numerical Binning-Less Isotonic Statistical Bivariate Numerical Modeling Method

“…Numerical data modeling is a broad research area which involves a wide variety of novel artificial-intelligence and statistical tools. A few examples of recent contributions in this field are Petri nets to model honeypot [13], extreme learning machines in monthly precipitation time series forecasting [14], and recurrent neural networks to language modeling, emotion classification and polyphonic modeling [15]. Specifically to the problem treated in this paper, traditional artificial neural-network techniques to build up empirical models of GFR were proposed and tested in the scientific literature, although their performance appeared unsatisfactory or questionable [16,17].…”

“…To conclude this section of the paper, let us consider a minimal working numerical example. Let D X = [2, 3, 5, 5, 6, 6, 7, 9] and D Y =[6,7,10,10,11,11,11,12,15,20], i.e., n = 8, m = 10. The actual underlying model is f (x) = 2x + 2.…”

A Non-Intrusive, Traffic-Aware Prediction Framework for Power Consumption in Data Center Operations

“…In this section, we evaluate the proposed prediction model, denoted as URL-LSTM, with other baseline algorithms, including Random Forest Regression (RF) [17], Support Vector Regression SVR [18], Long Short Term Memory (LSTM) [19] and Residual Recurrent Networks (RRN) [20]. We do not compare our proposed method with other power consumption prediction approaches in which intrusive features are needed.…”