Deep networks for system identification: a Survey

Abstract

Deep learning is a topic of considerable current interest. The availability of massive data collections and powerful software resources has led to an impressive amount of results in many application areas that reveal essential but hidden properties of the observations. System identification learns mathematical descriptions of dynamic systems from input-output data and can thus benefit from the advances of deep neural networks to enrich the possible range of models to choose from. For this reason, we provide a survey of deep learning from a system identification perspective. We cover a wide spectrum of topics to enable researchers to understand the methods, providing rigorous practical and theoretical insights into the benefits and challenges of using them. The main aim of the identified model is to predict new data from previous observations. This can be achieved with different deep learning based modelling techniques and we discuss architectures commonly adopted in the literature, like feedforward, convolutional, and recurrent networks. Their parameters have to be estimated from past data trying to optimize the prediction performance. For this purpose, we discuss a specific set of first-order optimization tools that is emerged as efficient. The survey then draws connections to the well-studied area of kernel-based methods. They control the data fit by regularization terms that penalize models not in line with prior assumptions. We illustrate how to cast them in deep architectures to obtain deep kernel-based methods. The success of deep learning also resulted in surprising empirical observations, like the counter-intuitive behaviour of models with many parameters. We discuss the role of overparameterized models, including their connection to kernels, as well as implicit regularization mechanisms which affect generalization, specifically the interesting phenomena of benign overfitting and double-descent. Finally, we highlight numerical, computational and software aspects in the area with the help of applied examples.1 This research has been partially supported by the Progetto di Ateneo CPDA147754/14-New statistical learning approach for multi-agents adaptive estimation and coverage control, the project Deep probabilistic regression -new models and learning algorithms (contract number: 2021-04301), funded by the Swedish Research Council, the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by Knut and Alice Wallenberg Foundation and by Kjell och Märta Beijer Foundation. This paper was not presented at any IFAC meeting. Corresponding author Gianluigi Pillonetto Ph.

Robust offset‐free nonlinear model predictive control for systems learned by neural nonlinear autoregressive exogenous models

“…where f px k q and gpx k q are the feed-forward NNs defined in ( 6) and (7), respectively. Let us then consider the following assumption.…”

“…(31) where gpx a q b u b has been summed and subtracted. Let us point out, at this stage, that both f p¨q and gp¨q are Lipschitzcontinuous, because such networks are defined as sequences of affine transformations followed by element-wise Lipschtizcontinuous activation functions, see ( 6) and (7). Therefore, it follows that }f px a,k q ´f px b,k q} 2 ď…”

“…To address the need of a reliable model, one can think to harness the modeling performance of Neural Networks (NNs) for nonlinear system identification tasks. In particular, among the wide variety of NN architectures nowadays available [7], Recurrent Neural Networks (RNNs) are considered to be the most suited for approximating dynamical systems [8]. Indeed, being stateful networks, RNNs can store information from the past data in their hidden states, which makes them particularly effective in learning time-series and dynamical systems [9], although they are notoriously hard to train [10].…”

“…Despite the positive results achieved in many control-related problems [7], RNNs are still treated with some caution by the control systems community due to the empirical nature of these tools, their difficult interpretability, and the lack of solid theoretical guarantees concerning the so-called generalization property, i.e. the capability to produce meaningful and consistent predictions even for data not included in the training set.…”