Recurrent Neural Network Training With Convex Loss and Regularization Functions by Extended Kalman Filtering

Physics-Informed Neural Network Modeling and Predictive Control of District Heating Systems

“…Thus, the performances of a standard GRU and of a physics-informed GRU (PI-GRU) model are now compared. These are trained with the 15 690-sample dataset over 1500 epochs, with the ADAM optimizer and a learning rate of 0.003, so as to get a good trade-off between convergence speed and excessive oscillations avoidance [6].…”

“…Please note that v = v [1] = v [2] = T s 0 , being the supply temperatures at nodes α 1 and α 2 directly affected by the one at the heating station node α 0 , which is the overall system input. Moreover, v [3] = y [1] s , being α 1 the only preceding significant node for α 3 , v [4] = v [5] = [y [2] ′ s , y [3] ′ s ] ′ , being α 2 and α 3 the preceding significant nodes for α 4 and α 5 , whereas u [6] = [y [1] ′ r , y [2] ′ r , y [3] ′ r , y [4] ′ r , y [5] ′ r ] ′ , since the return-associated RNN is fed with the load output temperatures and water flows which are outputs for the five load-associated RNNs.…”

“…To overcome these issues, it is here proposed to rely on identification methods with the purpose of obtaining computationally efficient and accurate models from operational data, which are typically widely available in DHSs. More specifically, recurrent neural networks (RNNs) are employed, being particularly suited to model nonlinear dynamical systems [6], [7]. It is worth remarking that RNNs generally do not exploit any physical insight on the identified system: this may lead to the need of large datasets, time-consuming training procedures, or even unreliable data-based models.…”

An Inexact Sequential Quadratic Programming Method for Learning and Control of Recurrent Neural Networks

“…Then, each of the step k ∈ K causes a sufficient decrease in M 1 by the backtracking line-search procedure. Moreover, with M 1 (z k ; ρ) ≥ K > −∞ for some constant K , the sufficient decrease condition (25) shows that…”

“…The second-order methods, however, can be used to mitigate these drawbacks by utilizing, in their learning procedures, the information contained in the curvature of the loss surface, which provably accelerates convergence. The second-order methods for learning RNNs have historically been used in two ways: 1) the use of second-order optimization algorithms, such as generalized Gauss-Newton (GGN), Levenberg-Marquardt (LM), and conjugate gradient (CG) algorithms (see [12], [13], [14], [15], [16], [17], [18], [19]) and 2) using nonlinear sequential stateestimation techniques, such as extended Kalman filter (EKF) method (see [20], [21], [22], [23], [24], [25]). In the first approach, the second-order information is captured through Hessian (or approximate Hessian) computations, while in the second approach, the second-order information is computed recursively as a prediction-error covariance matrix.…”