Interpretable Stochastic Model Predictive Control using Distributional Reinforced Estimation for Quadrotor Tracking Systems

Abstract

This paper presents a novel trajectory tracker for autonomous quadrotor navigation in dynamic and complex environments. The proposed framework integrates a distributional Reinforcement Learning (RL) estimator for unknown aerodynamic effects into a Stochastic Model Predictive Controller (SMPC) for trajectory tracking. Aerodynamic effects derived from drag forces and moment variations are difficult to model directly and accurately. Most current quadrotor tracking systems therefore treat them as simple 'disturbances' in conventional control approaches. We propose Quantile-approximationbased Distributional Reinforced-disturbance-estimator, an aerodynamic disturbance estimator, to accurately identify disturbances, i.e., uncertainties between the true and estimated values of aerodynamic effects. Simplified Affine Disturbance Feedback is employed for control parameterization to guarantee convexity, which we then integrate with a SMPC to achieve sufficient and non-conservative control signals. We demonstrate our system to improve the cumulative tracking errors by at least 66% with unknown and diverse aerodynamic forces compared with recent state-of-the-art. Concerning traditional Reinforcement Learning's non-interpretability, we provide convergence and stability guarantees of Distributional RL and SMPC, respectively, with non-zero mean disturbances.

QuaDUE-CCM: Interpretable Distributional Reinforcement Learning using Uncertain Contraction Metrics for Precise Quadrotor Trajectory Tracking

“…While neural network-based approaches learn the nonlinear dynamical effects more accurately [13], achieving adaptability and robustness from these learning-based approaches are still ongoing challenges. One important reason is that these training datasets are collected from simulation and/or real-world historical records, from which it is difficult to fully describe the complexity in their environments [14].…”

“…Control Contraction Certificates: most existing methods on quadrotor trajectory tracking have so far been demonstrated in stabilization, i.e., a fixed-point-tracking controller [29,30,14]. These fixed-point-tracking controllers are relatively efficient in a low-speed and low-variance trajectory.…”

“…Therefore, we present the convergence analysis of QuaDUE for Policy Evaluation and Policy Improvement, respectively. Proposition 2 (Policy Evaluation [38,14]): Given a deterministic policy π, a quantile approximator Π W1 and Z k+1 (s, a) = Π W1 T π Z k (s, a), the sequence Z k (s, a) converges to a unique fixed point…”

“…Proposition 3 (Policy Improvement [14]): Denoting an old policy by π old and a new policy by π new , there exists E[Z(s, a)] πnew (s, a) ≥ E[Z(s, a)] π old (s, a), ∀s ∈ S and ∀a ∈ A.…”

“…Similar to [14], we denote J θ (s, a) = D KL (p s,a , q s,a θ ) as the histogram distributional loss, where p s,a (x) and q s,a θ are the true and approximated density function of Z(s, a). We denote E J (θ) = E (s,a)∼ρπ [J θ (s, a)] is the expectation of J θ , where ρ π is the generated distribution under the deterministic policy π.…”