Ultra-Orthogonal Forward Regression Algorithms for the Identification of Non-Linear Dynamic Systems

Abstract

A new Ultra Least Squares (ULS) criterion is introduced for system identification. Unlike the standard least squares criterion which is based on the Euclidean norm of the residuals, the new ULS criterion is derived from the Sobolev space norm. The new criterion measures not only the discrepancy between the observed signals and the model prediction but also the discrepancy between the associated weak derivatives of the observed and the model signals. The new ULS criterion possesses a clear physical interpretation and is easy to implement. Based on this, a new Ultra Orthogonal Forward Regression (UOFR) algorithm is introduced for nonlinear system identification, which includes converting a least squares regression problem into the associated ultra least squares problem and solving the ultra least squares problem using the orthogonal forward regression method. Numerical simulations show that the new UOFR algorithm can significantly improve the performance of the classic OFR algorithm.Key words: orthogonal forward regression, system identification, ultra least squares, ultra orthogonal forward regression, ultra orthogonal least squares. IntroductionSystem identification plays a more and more important role in revealing the unknown mechanisms and rules underlying complex phenomena (Schmidt & Lipson, 2009). System identification includes the detection of the model structure and estimation of the associated parameters. A system identification problem can often be thought of as an optimization problem where the optimal model is searched from a large predefined candidate model dictionary given a criterion. The criterion is 2 used to evaluate the performance of each model by measuring the discrepancy between the observed data and the model predictions. The candidate model dictionary is often chosen to be large enough to include the unknown correct model. Hence an exhaustive search algorithm is often infeasible in these kinds of applications because of the large solution space. Even an evolutionary algorithm which can greatly reduce the search process can still be very computationally intensive.Hence an algorithm which can efficiently find the optimal solution is desired. However, a fast algorithm often dictates an optimal substructure; otherwise the search may converge to a suboptimal solution. Many efforts have been made to improve the search process under a certain specific loss function or performance index, for example, the simulated annealing algorithm, particle swarm optimisation, and so on. In this paper, a different and new methodology will be introduced.Instead of improving the search method, a new and effective criterion will be introduced to describe the objective of the regression more accurately. Under the new criterion, the solution space has a better structure and a fast algorithm is more likely to find the optimal solution.System identification aims to identify a model from observed data based on a criterion. A good criterion results in not only better parameter estimation but also a good search pa...

Orthogonal Floating Search Algorithms: From the perspective of nonlinear system identification

“…From the search perspective, the OFR is a sequential greedy approach where the term with the highest performance metric, referred to as Error-Reduction-Ratio (ERR), is included in each step and it has proven to be very effective in various applications [3,11]. However, it has been shown that under certain conditions the OFR may yield sub-optimal term subsets [12][13][14][15][16]. The reason for such suboptimal performance is often attributed to the performance metric, ERR.…”

Modeling and prediction of global magnetic disturbance in near-Earth space: A case study for K p index using NARX models

“…In the literature, some authors have adapted the original OFR algorithm to optimize directly the MPO in order to obtain a better long‐term prediction. However, these modified versions tend to be computationally expensive during the feature selection step, and a much better alternative is to use the iterative or ultraorthogonalization approach [ Guo et al , , ].…”

“…However, because the NARX model (1) depends on past outputs, a more reliable way to check the validity of the model is through the model-predicted output (MPO), which uses past predicted outputs to estimate future ones and to provide details about the stability and predictability range of the model, In the literature, some authors have adapted the original OFR algorithm to optimize directly the MPO in order to obtain a better long-term prediction. However, these modified versions tend to be computationally expensive during the feature selection step, and a much better alternative is to use the iterative or ultraorthogonalization approach [Guo et al, 2015a[Guo et al, , 2015b].…”

“…In recent years, several variants have been proposed that improve the performance of the original NARX algorithm. Such variations include the use of more complex and flexible predefined functions such as wavelets [ Alexandridis and Zapranis , ; Billings and Wei , , ], radial basis functions [ Billings et al , ; Wei et al , , ], and ridge basis functions [ Wei et al , ], together with an improved search mechanism such as the common model structure selection [ Wei and Billings , ; Li et al , , ], iterative search [ Guo et al , ], incorporation of weak derivatives information [ Guo et al , ], and other dependency metrics [ Koller and Sahami , ; Billings and Wei , ; Wei and Billings , ; Wang et al , ; Speed , ; Reshef et al , ; Székely et al , ; Székely and Rizzo , ; Piroddi and Spinelli , ; Ayala Solares and Wei , ].…”

Multi-objective evolutionary framework for non-linear system identification: A comprehensive investigation

“…Over the years, this notion of the original OFR approach has been refined to further improve the search performance. These include OFR approaches with either new term evaluation criteria [17][18][19][20] or improved search strategies [21][22][23][24][25]. A detailed treatment on this subject can be found in the recent investigation by the authors [25].…”