Adaptive Stress Testing of Trajectory Predictions in Flight Management Systems

Abstract

To find failure events and their likelihoods in flightcritical systems, we investigate the use of an advanced blackbox stress testing approach called adaptive stress testing. We analyze a trajectory predictor from a developmental commercial flight management system which takes as input a collection of lateral waypoints and en-route environmental conditions. Our aim is to search for failure events relating to inconsistencies in the predicted lateral trajectories. The intention of this work is to find likely failures and report them back to the developers so they can address and potentially resolve shortcomings of the system before deployment. To improve search performance, this work extends the adaptive stress testing formulation to be applied more generally to sequential decision-making problems with episodic reward by collecting the state transitions during the search and evaluating at the end of the simulated rollout. We use a modified Monte Carlo tree search algorithm with progressive widening as our adversarial reinforcement learner. The performance is compared to direct Monte Carlo simulations and to the crossentropy method as an alternative importance sampling baseline. The goal is to find potential problems otherwise not found by traditional requirements-based testing. Results indicate that our adaptive stress testing approach finds more failures and finds failures with higher likelihood relative to the baseline approaches.

A Survey of Algorithms for Black-Box Safety Validation of Cyber-Physical Systems

“…Monte Carlo tree search (MCTS) is an online planning algorithm for sequential decision making that has seen success for long time-horizon problems (Silver et al, 2016), making it useful for safety validation (Delmas et al, 2019;Julian et al, 2020;Lee et al, 2020;Moss et al, 2020;Wicker et al, 2018;Zhang et al, 2018). Monte Carlo tree search uses online planning to determine the best actions to take from a starting state to maximize reward.…”

“…Flight Path-Planning. Systems are given a mission, which is a series of waypoints that the system must navigate (Moss et al, 2020;Tuncali et al, 2018), or a target location and keep-out zones along the path (Lee et al, 2019). Falsifying trajectories may be different mission parameters or disturbances during mission execution, such as wind and sensor noise.…”

“…5 The AST Toolbox is open-source and available under the MIT license. 6 Two related toolboxes, called AdaptiveStressTesting.jl (Lee et al, 2020) and POMDP-StressTesting.jl (Moss et al, 2020), follow a similar paradigm as the AST Toolbox but are implemented in the Julia programming language (Bezanson et al, 2017). Julia can interface directly with many other programming languages, 7 and notably, Julia can interface with MATLAB through the MATLAB.jl package.…”

POMDPStressTesting.jl: Adaptive Stress Testing for Black-Box Systems

“…POMDPStressTesting.jl has been used to find likely failures in aircraft trajectory prediction systems (Moss et al, 2020), which are flight-critical subsystems used to aid in-flight automation. A developmental commercial flight management system was stress tested so the system engineers could mitigate potential issues before system deployment (Moss et al, 2020). In addition to traditional requirements-based testing for avionics certification (RTCA, 2011), this work is being used to find potential problems during development.…”

“…

POMDPStressTesting.jl is a package that uses reinforcement learning and stochastic optimization to find likely failures in black-box systems through a technique called adaptive stress testing (Lee et al, 2019). Adaptive stress testing (AST) has been used to find failures in safety-critical systems such as aircraft collision avoidance systems (Lee et al, 2015), flight management systems (Moss et al, 2020), and autonomous vehicles (Koren et al, 2018). The POMDPStressTesting.jl package is written in Julia (Bezanson et al, 2017) and is part of the wider POMDPs.jl ecosystem (Egorov et al, 2017), which provides access to simulation tools, policies, visualizations, and-most importantly-solvers.

…”