A Study of Aero-Engine Control Method Based on Deep Reinforcement Learning

Abstract

A novel aero-engine control method based on deep reinforcement learning (DRL) is proposed to improve the engine response ability. The Q-learning that is model free and can be performed online is adopted. For improving the learning capacity of DRL, the online sliding window deep neural network (OL-SW-DNN) is proposed and adopted to estimate the action value function. The OL-SW-DNN selects the nearest point data with certain length as training data and is insensitivity to the noise. Finally, the comparison simulations of the proposed method with the proportion-integration-differentiation (PID) that is the most commonly used as an engine controller algorithm in industry are conducted to verify the validity of the proposed method. The results show that, compared with the PID, the acceleration time of the proposed method decreased by 1.525 s under the premise of satisfying all engine limits. INDEX TERMS Aero-engine control method, response ability, deep reinforcement learning, on line, deep neural network.

Aero-engine acceleration control using deep reinforcement learning with phase-based reward function

“…In this study, the conventional aero-engine acceleration control task is formulated into Aero-engine Acceleration Control Markov Decision Process (AACMDP) problem, which is suitable for DRL algorithms to solve. This study extends prior work 13 by applying the TRPO with phase-based reward function to the aero-engine acceleration control. In addition, this study provides a new idea for designing the reward function, which is used by DRL in feedback control tasks.…”

“…Figure 1 presents the traditional control structure of aero-engine based on PID controller. As described in Ref 13,. this control structure contains PID controller, acceleration schedule functions, deceleration schedule functions, acceleration limits unit, deceleration limits unit, MIN and MAX selector, and actuator and aero-engine.…”

“…The acceleration/deceleration schedule functions represent the acceleration/deceleration control law of the aero-engine. They work together with the PID controller in the aero-engine acceleration/deceleration control.

Figure 1.Traditional control structure of aero-engine based on PID controller 13

…”

“…As described in Ref. 13, this control structure contains PID controller, acceleration schedule functions, deceleration schedule functions, acceleration limits unit, deceleration limits unit, MIN and MAX selector, and actuator and aero-engine. There are many limits during the operation of aero-engine, such as rotor speed limits of fan and compressor, the temperature limit of turbine inlet temperature, and surge margin limits of fan and compressor.…”

“…It is a modified version of the one proposed in Ref. 13. In our study, engine states are also passed to the DRL controller.…”