Reinforcement-Learning-Based Dual-Control Methodology for Complex Nonlinear Discrete-Time Systems With Application to Spark Engine EGR Operation

Abstract

A novel reinforcement-learning-based dual-control methodology adaptive neural network (NN) controller is developed to deliver a desired tracking performance for a class of complex feedback nonlinear discrete-time systems, which consists of a second-order nonlinear discrete-time system in nonstrict feedback form and an affine nonlinear discrete-time system, in the presence of bounded and unknown disturbances. For example, the exhaust gas recirculation (EGR) operation of a spark ignition (SI) engine is modeled by using such a complex nonlinear discrete-time system. A dual-controller approach is undertaken where primary adaptive critic NN controller is designed for the nonstrict feedback nonlinear discrete-time system whereas the secondary one for the affine nonlinear discrete-time system but the controllers together offer the desired performance. The primary adaptive critic NN controller includes an NN observer for estimating the states and output, an NN critic, and two action NNs for generating virtual control and actual control inputs for the nonstrict feedback nonlinear discrete-time system, whereas an additional critic NN and an action NN are included for the affine nonlinear discrete-time system by assuming the state availability. All NN weights adapt online towards minimization of a certain performance index, utilizing gradient-descent-based rule. Using Lyapunov theory, the uniformly ultimate boundedness (UUB) of the closed-loop tracking error, weight estimates, and observer estimates are shown. The adaptive critic NN controller performance is evaluated on an SI engine operating with high EGR levels where the controller objective is to reduce cyclic dispersion in heat release while minimizing fuel intake. Simulation and experimental results indicate that engine out emissions drop significantly at 20% EGR due to reduction in dispersion in heat release thus verifying the dual-control approach.

Deep reinforcement learning for dynamic control of fuel injection timing in multi-pulse compression ignition engines

“…50,51 Several authors have explored combustion control strategies using traditional RL for hard-coal combustion processes in a power plant, 52,53 for spark ignition and injection timing, 54,55 for energy management strategies for hybrid-electric vehicles, [56][57][58] and for the control of the air-fuel ratio 59 and spark engine EGR operation. 60 This past research has focused on the use of MDPs to model the problem and Q-learning techniques to solve the control problem. These traditional techniques have shown to be not as expressive as DRL, which leverages the expressiveness of deep neural networks to tackle complex control challenges.…”

Online optimal control for VTOL aircraft system based on DHP algorithm

“…In the past decades, optimal control problems have been caused widely attention in the control community. ADP has now been a very promising framework to solve learning optimal control problems up to now [1][2][3][4][5][6][7][8]. Based on ADP, [9] design a near ¿nite-horizon optimal controller with ε error bound for discrete-time nonlinear systems.…”

Optimal control for discrete-time affine non-linear systems using general value iteration

“…In recent years, RL has been applied to feedback control [17][18][19][20][21][22][23]. On the other hand, iterative learning control can be seen in [24,25].…”