Model-Free Multi-Kernel Learning Control for Nonlinear Discrete-Time Systems

Abstract

Reinforcement learning (RL) has become an important research topic to solve learning control problems of nonlinear dynamic systems. In RL, feature representation is a critical factor for improving the performance of online or offline learning controllers. Although multikernel learning has been studied in supervised learning problems, there is little work on multi-kernel-based feature representation in RL algorithms. In this paper, a model-free multi-kernel learning control (MMLC) approach is proposed for a class of nonlinear discrete-time systems. MMLC has advantages over other singlekernel-based RL algorithms in that the parameters in the kernel functions can be learned adaptively. Furthermore, MMLC uses a model-free actor-critic learning structure, where the critic is designed to approximate the derivatives of value functions. Different from the popularly studied dual heuristical programming algorithm, the proposed MMLC approach can learn the dynamics of the nonlinear system in a data-driven way. To evaluate the performance of MMLC, single-link and double-link inverted pendulums are employed as two benchmarks. The effectiveness of the MMLC algorithm has been demonstrated in simulation. It is shown that MMLC can achieve better performance than previous kernel-based dual heuristic programming with partial model information.

Synchronous multi‐kernel iterative dual heuristic programming with ℓ2‐regularisation

“…Following the same line with [18], setting , it is possible to compute the optimal control policy at time k as a function of the next state where is the costate that can be obtained as follows: As the system given by (1) is non‐linear, it is difficult to analytically calculate that needs to be used in (4). To solve this problem, kernel‐based least‐squares iterative method has been widely used for policy and value evaluation in the framework of DHP.…”

“…Compared with single‐kernel designs, the adopted multi‐kernel structure consists of a linear combination of weighted single‐kernel functions, hence it is capable in reducing the complexity of kernel width parameter tuning for feature representation, especially for high dimensional and heterogeneous data‐samples. Furthermore, the multi‐kernel based feature representation is both used for optimal policy and VFA, which is different from the work in [18].…”

“…In [17], an effective MKL method has been proposed, which can achieve an autonomous selection of models for classification and regression problems and extract understandable information of learning problems. An incremental DHP with multiple kernel functions has been presented in [18] for the optimal control of discrete‐time non‐linear systems. Although MKL has successfully been applied to many practical tasks, there are still few research works on the approximation of multi‐kernel structure both in the actor module and the critic module.…”