Reinforcement-Learning-Based Output-Feedback Control of Nonstrict Nonlinear Discrete-Time Systems With Application to Engine Emission Control

Shih, Peter; Kaul, Brian; Jagannathan, S.; Drallmeier, Joseph A.

doi:10.1109/tsmcb.2009.2013272

Cited by 34 publications

(5 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This approach was later improved by a reinforcement learning (RL)-based controller that used an ANN-based adaptive-critic structure. 26,27 All these applications, however, were trained offline using the model of Daw et al 28 While they showed improvements in CCV, they also exhibited fuel enrichment, and it was not possible to determine how much of the improvement was due to next-cycle control actions. Even though the model has been improved 29 and recent model-based controllers have been designed, 30,31 offline training suffers from inaccuracies and uncertainties of the model.…”

Section: Introductionmentioning

confidence: 99%

Reinforcement learning applied to dilute combustion control for increased fuel efficiency

Maldonado,

Kaul,

Schuman

et al. 2024

International Journal of Engine Research

View full text Add to dashboard Cite

To reduce the modeling burden for control of spark-ignition engines, reinforcement learning (RL) has been applied to solve the dilute combustion limit problem. Q-learning was used to identify an optimal control policy to adjust the fuel injection quantity in each combustion cycle. A physics-based model was used to determine the relevant states of the system used for training the control policy in a data-efficient manner. The cost function was chosen such that high cycle-to-cycle variability (CCV) at the dilute limit was minimized while maintaining stoichiometric combustion as much as possible. Experimental results demonstrated a reduction of CCV after the training period with slightly lean combustion, contributing to a net increase in fuel conversion efficiency of 1.33%. To ensure stoichiometric combustion for three-way catalyst compatibility, a second feedback loop based on an exhaust oxygen sensor was incorporated into the fuel quantity controller using a slow proportional-integral (PI) controller. The closed-loop experiments showed that both feedback loops can cooperate effectively, maintaining stoichiometric combustion while reducing combustion CCV and increasing fuel conversion efficiency by 1.09%. Finally, a modified cost function was proposed to ensure stoichiometric combustion with a single controller. In addition, the learning period was shortened by half to evaluate the RL algorithm performance on limited training time. Experimental results showed that the modified cost function could achieve the desired CCV targets, however, the learning time was reduced by half and the fuel conversion efficiency increased only by 0.30%.

show abstract

Section: Introductionmentioning

confidence: 99%

Reinforcement learning applied to dilute combustion control for increased fuel efficiency

Maldonado,

Kaul,

Schuman

et al. 2024

International Journal of Engine Research

View full text Add to dashboard Cite

show abstract

“…30,31 RL has been used for automotive powertrain control systems especially in energy management of hybrid electric vehicles [32][33][34] and for internal combustion engines. [35][36][37][38][39][40] Q-learning RL is used as idle speed control of a spark-ignition (SI) engine by controlling the spark timing and intake throttle valve position. 41 Similar studies have been carried out for diesel engine idle speed control by controlling the fuel injection timing.…”

Section: Introductionmentioning

confidence: 99%

“… 36 RL has also been used for emission control of SI engines. 37 , 38 A very limited number of studies have been carried out utilizing RL for internal combustion control, and most of the existing work has focused on SI engines. To the authors’ knowledge, deep RL algorithms have not been previously implemented for diesel engine performance and emission control.…”

Section: Introductionmentioning

confidence: 99%

Safe deep reinforcement learning in diesel engine emission control

Norouzi

Shahpouri

Gordon

et al. 2023

Proceedings of the Institution of Mechanical Engineers, Part I:

View full text Add to dashboard Cite

A deep reinforcement learning application is investigated to control the emissions of a compression ignition diesel engine. The main purpose of this study is to reduce the engine-out nitrogen oxide [Formula: see text] emissions and to minimize fuel consumption while tracking a reference engine load. First, a physics-based engine simulation model is developed in GT-Power and calibrated using experimental data. Using this model and a GT-Power/Simulink co-simulation, a deep deterministic policy gradient is developed. To reduce the risk of an unwanted output, a safety filter is added to the deep reinforcement learning. Based on the simulation results, this filter has no effect on the final trained deep reinforcement learning; however, during the training process, it is crucial to enforce constraints on the controller output. The developed safe reinforcement learning is then compared with an iterative learning controller and a deep neural network–based nonlinear model predictive controller. This comparison shows that the safe reinforcement learning is capable of accurately tracking an arbitrary reference input while the iterative learning controller is limited to a repetitive reference. The comparison between the nonlinear model predictive control and reinforcement learning indicates that for this case reinforcement learning is able to learn the optimal control output directly from the experiment without the need for a model. However, to enforce output constraint for safe learning reinforcement learning, a simple model of system is required. In this work, reinforcement learning was able to reduce [Formula: see text] emissions more than the nonlinear model predictive control; however, it suffered from slightly higher error in load tracking and a higher fuel consumption.

show abstract

“…Zhang et al (2022) discussed the model reference control (MRC) issue for a class of discrete-time linear systems, where the MRC feedback control law was designed only using the reference input and quantized output signals. In the work by Shih et al (2009), by proposing a dynamic state observer, an RL-based output feedback controller was designed to guarantee the desired tracking performance for some uncertain nonlinear discrete-time systems.…”

Section: Introductionmentioning

confidence: 99%

Reinforcement learning event-triggered output feedback control for uncertain nonlinear discrete systems

Ren,

Li,

Song

2023

Transactions of the Institute of Measurement and Control

View full text Add to dashboard Cite

In this paper, a novel reinforcement learning (RL)-based event-triggered (ET) output feedback control algorithm is proposed for a class of uncertain strict-feedback nonlinear discrete-time systems. In contrast to traditional RL-based control methods, we proposed an ET output feedback controller based on the backstepping technique, where the transmission cost can be efficiently conserved. Then, in light of the radial basis function (RBF) neural network (NN), various critic NNs are constructed to approximate the critic functions in each step. Furthermore, with the backing of the proposed ET mechanism, a sampled output feedback controller is addressed to guarantee that the tracking errors and all signals of the closed-loop system are semi-global uniformly ultimately bounded (SGUUB). Finally, a simulation example is presented to demonstrate the effectiveness of the control strategy.

show abstract

Reinforcement-Learning-Based Output-Feedback Control of Nonstrict Nonlinear Discrete-Time Systems With Application to Engine Emission Control

Cited by 34 publications

References 11 publications

Reinforcement learning applied to dilute combustion control for increased fuel efficiency

Reinforcement learning applied to dilute combustion control for increased fuel efficiency

Safe deep reinforcement learning in diesel engine emission control

Reinforcement learning event-triggered output feedback control for uncertain nonlinear discrete systems

Contact Info

Product

Resources

About