Control capability: From two-loop autopilot to three-loop autopilot

Two-Loop Acceleration Autopilot Design and Analysis Based on TD3 Strategy

“…At the same time, the flight state s t determined the control parameters k A and k g through the actor-network and exploration noise ϵ and output the action a t . Then, the system stability margin is calculated according to Equation ( 6) and Equation ( 26), and the real-time reward r t is calculated according to Equation (19).…”

“…In order to solve the strong coupling problem between the adaptive updating loop and the strict-feedback control loop in traditional control, a low-frequency learning structure consisting of a low-pass filter, a state estimator, and a multilayer perceptron (MLP) neural network was proposed in [18], which reduced the computation complexity and effectively avoided the update explosion problem. Literature [19] designed a two-loop autopilot based on a missile longitudinal channel model employing the RL principle and solved the quadratic optimal tracking control problem employing an actor-critic structure. The results demonstrated that the designed controller possessed an excellent tracking effect and dynamic performance.…”

“…The results demonstrated that the designed controller possessed an excellent tracking effect and dynamic performance. Different from literature [19], literature [20] transformed the aircraft control procedure into a Markov decision procedure. Utilizing the deep deterministic policy gradient (DDPG) algorithm [21], the optimal PID controller control parameters were determined iteratively.…”

“…In addition, literature [25] made a comparative analysis of PID neural network controller and DDPG controller and also provided a conceptual proof that reinforcement learning can effectively solve the adaptive optimal control problem of nonlinear dynamic systems. However, most of the methods in [19,20,22,24] and [25] were designed and analyzed based on nonglobal profile state rather than entire flight envelope, which led to insufficient consideration of global constraints and performance indicators. When there was great uncertainty in the flight environment, the autonomy and robustness of the nominal trajectory tracking guidance mode were poor.…”

“…(1) Contrasting to the previous alternatives [19,20,22,24,25] based on nonglobal profile state, herein the proposed method takes the entire flight envelope state space as the research object. The TD3 algorithm is used to offline learn the control parameters for the entire flight envelope, which can ensure the autopilot has good performance during the flight envelope.…”