Closed-Loop Q-Learning Control of a Small Unmanned Aircraft

Clarke, R.J.; Fletcher, Liam; Greatwood, Colin; Waldock, Antony; Richardson, Thomas

doi:10.2514/6.2020-1234

Cited by 8 publications

(8 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Discrete actions spaces have been used previously with both Deep Q-Network [6] and PPO [7]. The actions are encoded as a pair of arrays of angular rates for the wing sweep and elevator, with the action effectively selecting a pair of indices into these arrays.…”

Section: Continuous Actionsmentioning

confidence: 99%

“…Model-based RL tends to be more sample efficient than model-free, however it is more complex and challenging to train [14]. Clarke et al used a Deep Q-Network (DQN), a model-free, value-based algorithm [6]. DQN, developed by Mnih et al, was the first deep-RL algorithm to be successfully demonstrated.…”

Section: Reinforcement Learningmentioning

confidence: 99%

“…RL was used as a schedule optimiser in an open-loop manner, where the DQN algorithm was used to pre-generate a schedule of actions [1]. Clarke et al trained an RL controller in simulation, then deployed it on the aircraft to form a closedloop RL controller [6]. The aircraft state from the flight controller was sent to the deployed network to select an action during flight.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Improvements in learning to control perched landings

et al. 2022

View full text Add to dashboard Cite

Reinforcement learning has previously been applied to the problem of controlling a perched landing manoeuvre for a custom sweep-wing aircraft. Previous work showed that the use of domain randomisation to train with atmospheric disturbances improved the real-world performance of the controllers, leading to increased reward. This paper builds on the previous project, investigating enhancements and modifications to the learning process to further improve performance, and reduce final state error. These changes include modifying the observation by adding information about the airspeed to the standard aircraft state vector, employing further domain randomisation of the simulator, optimising the underlying RL algorithm and network structure, and changing to a continuous action space. Simulated investigations identified hyperparameter optimisation as achieving the most significant increase in reward performance. Several test cases were explored to identify the best combination of enhancements. Flight testing was performed, comparing a baseline model against some of the best performing test cases from simulation. Generally, test cases that performed better than the baseline in simulation also performed better in the real world. However, flight tests also identified limitations with the current numerical model. For some models, the chosen policy performs well in simulation yet stalls prematurely in reality, a problem known as the reality gap.

show abstract

Section: Continuous Actionsmentioning

confidence: 99%

Section: Reinforcement Learningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Improvements in learning to control perched landings

et al. 2022

View full text Add to dashboard Cite

show abstract

“…In this work, Deep Deterministic Gradient Policy (DDGP), Trust Region Policy Optimisation (TRPO) and Proximal Policy Optimisation (PPO), were trained and compared in simulation and the PPO algorithm showed superior performance overall, including outperforming a conventional controller. Fixed-wing UAVs have also seen RL algorithms used for control, showing high levels of performance for attitude control in simulation [38], for angle of attack control in wind tunnel tests [25], or for perched landing [39][40][41]. A Q-learning RL algorithm was used to attain a policy based agent to find the best flight profile for perching manoeuvre in open loop flight tests [39].…”

Section: Introductionmentioning

confidence: 99%

“…A Q-learning RL algorithm was used to attain a policy based agent to find the best flight profile for perching manoeuvre in open loop flight tests [39]. Flight tests of the same platform in closed-loop configuration [40] followed on from this, where a Deep Q-Network (DQN) algorithm controlled the aircraft during flight but suffered from the reality gap between the simulation environment it had learnt from and flight tests. More recently, flight tests showed a reduced reality gap when an improved version of this controller was developed using PPO [41].…”

Section: Introductionmentioning

confidence: 99%

Reinforcement Learning to Control Lift Coefficient Using Distributed Sensors on a Wind Tunnel Model

Guerra-Langan

Araujo-Estrada

Windsor

2022

AIAA SCITECH 2022 Forum

View full text Add to dashboard Cite

Arrays of sensors distributed on the wing of fixed-wing vehicles can provide information not directly available to conventional sensor suites. These arrays of sensors have the potential to improve flight control and overall flight performance of small fixed-wing uninhabited aerial vehicles (UAVs). This work investigated the feasibility of estimating and controlling aerodynamic coefficients using the experimental readings of distributed pressure and strain sensors across a wing. The study was performed on a one degree-of-freedom model about pitch of a fixed-wing platform instrumented with the distributed sensing system. A series of reinforcement learning (RL) agents were trained in simulation for lift coefficient control, then validated in wind tunnel experiments. The performance of RL-based controllers with different sets of inputs in the observation space were compared with each other and with that of a manually tuned PID controller. Results showed that hybrid RL agents that used both distributed sensing data and conventional sensors performed best across the different tests.

show abstract

Event-Based Guidance and Incremental Control with Application to Fixed-wing Unmanned Aerial Vehicle Perched Landing Maneuvers

Song,

Sun,

Tao

et al. 2024

J Intell Robot Syst

View full text Add to dashboard Cite

Considering the nonlinearity and unknown dynamics of fixed-wing unmanned aerial vehicles in perched landing maneuvers, an event-based online guidance and incremental control scheme is proposed. The guidance trajectory for perched landing must be dynamically feasible therefore an event-based trapezoidal collocation point optimization method is proposed. Introduction of the triggering mechanism for the rational use of computing resources to improve PL accuracy. Furthermore, a filter-based incremental nonlinear dynamic inverse (F-INDI) control with state transformation is proposed to achieve robust trajectory tracking under high angle of attack (AOA). The F-INDI uses low-pass filters to obtain incremental dynamics of the system, which simplifies the design process. The state transformation strategy is to convert the flight-path angle, AOA and velocity into two composite dynamics, which avoids the sign reversal problem of control gain under high AOA. The stability analysis shows that the original states can be controlled only by controlling the composite state. Simulation results show that the proposed scheme achieves high perched landing accuracy and a reliable trajectory tracking control.

show abstract

Closed-Loop Q-Learning Control of a Small Unmanned Aircraft

Cited by 8 publications

References 7 publications

Improvements in learning to control perched landings

Improvements in learning to control perched landings

Reinforcement Learning to Control Lift Coefficient Using Distributed Sensors on a Wind Tunnel Model

Event-Based Guidance and Incremental Control with Application to Fixed-wing Unmanned Aerial Vehicle Perched Landing Maneuvers

Contact Info

Product

Resources

About