Adaptive Stress Testing for Autonomous Vehicles

Koren, Mark; Alsaif, Saud; Lee, Ritchie; Kochenderfer, Mykel J.

doi:10.1109/ivs.2018.8500400

Cited by 141 publications

(128 citation statements)

References 16 publications

Supporting

Mentioning

128

Contrasting

Order By: Relevance

“…This section outlines the problem used in simulation to test AST, the hyper-parameters of the DRL solver, and the reward structure. For bench-marking purposes, we follow the experiment setup-simulation, pedestrian models, and SUT model-proposed in our previous work [10]. The problem has a 5-dimensional state-space and a 6-dimensional action space, and is run for up to 50 time-steps.…”

Section: Methodsmentioning

confidence: 99%

“…We previously added a new deep reinforcement learning (DRL) solver to AST [10]. The solver is interchangeable with the commonly-used MCTS solver.…”

Section: B Recurrent Deep Reinforcement Learning Solvermentioning

confidence: 99%

“…Reinforcement learning techniques can be used to solve the MDP, with the reward function depending on the likelihood of actions taken and whether a failure was found. We recently introduced a deep reinforcement learning (DRL) solver that was able to find failures in an example autonomous vehicle scenario more efficiently than an existing Monte Carlo tree search (MCTS) solver [10]. However, there are two major limitations that make using the solver more challenging: the solver's dependence on the simulation state and requirement to be run from a single initial condition.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Efficient Autonomy Validation in Simulation with Adaptive Stress Testing

Koren

Kochenderfer

2019

2019 IEEE Intelligent Transportation Systems Conference (ITSC)

Self Cite

View full text Add to dashboard Cite

During the development of autonomous systems such as driverless cars, it is important to characterize the scenarios that are most likely to result in failure. Adaptive Stress Testing (AST) provides a way to search for the mostlikely failure scenario as a Markov decision process (MDP). Our previous work used a deep reinforcement learning (DRL) solver to identify likely failure scenarios. However, the solver's use of a feed-forward neural network with a discretized space of possible initial conditions poses two major problems. First, the system is not treated as a black box, in that it requires analyzing the internal state of the system, which leads to considerable implementation complexities. Second, in order to simulate realistic settings, a new instance of the solver needs to be run for each initial condition. Running a new solver for each initial condition not only significantly increases the computational complexity, but also disregards the underlying relationship between similar initial conditions. We provide a solution to both problems by employing a recurrent neural network that takes a set of initial conditions from a continuous space as input. This approach enables robust and efficient detection of failures because the solution generalizes across the entire space of initial conditions. By simulating an instance where an autonomous car drives while a pedestrian is crossing a road, we demonstrate the solver is now capable of finding solutions for problems that would have previously been intractable.

show abstract

Section: Methodsmentioning

confidence: 99%

“…We previously added a new deep reinforcement learning (DRL) solver to AST [10]. The solver is interchangeable with the commonly-used MCTS solver.…”

Section: B Recurrent Deep Reinforcement Learning Solvermentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Efficient Autonomy Validation in Simulation with Adaptive Stress Testing

Koren

Kochenderfer

2019

2019 IEEE Intelligent Transportation Systems Conference (ITSC)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The constants α and β are set to 10 000 and 1000, respectively, to penalize the algorithm for not finding a collision. Solvers used in our application include Monte Carlo Tree Search (MCTS) [14] and Trust Region Policy Optimization (TRPO) [15] because both have been shown to successfully find failures when combined with AST [8], [11].…”

Section: A Adaptive Stress Testingmentioning

confidence: 99%

Adaptive Stress Testing with Reward Augmentation for Autonomous Vehicle Validatio

Corso

Driggs-Campbell

et al. 2019

2019 IEEE Intelligent Transportation Systems Conference (ITSC)

Self Cite

View full text Add to dashboard Cite

Determining possible failure scenarios is a critical step in the evaluation of autonomous vehicle systems. Real world vehicle testing is commonly employed for autonomous vehicle validation, but the costs and time requirements are high. Consequently, simulation driven methods such as Adaptive Stress Testing (AST) have been proposed to aid in validation. AST formulates the problem of finding the most likely failure scenarios as a Markov decision process, which can be solved using reinforcement learning. In practice, AST tends to find scenarios where failure is unavoidable and tends to repeatedly discover the same types of failures of a system. This work addresses these issues by encoding domain relevant information into the search procedure. With this modification, the AST method discovers a larger and more expressive subset of the failure space when compared to the original AST formulation. We show that our approach is able to identify useful failure scenarios of an autonomous vehicle policy.

show abstract

“…Koren et al [50] used DRL for adaptive stress testing of autonomous vehicles aimed at finding some problematic selfdriving scenarios which may lead to a collision with a moving pedestrian. Their work is similar to our previous work in [51] that finds the worst sequences of actions to maximize the resource utilization on the SUT.…”

Section: Related Workmentioning

confidence: 99%

Using Deep Reinforcement Learning for Exploratory Performance Testing of Software Systems With Multi-Dimensional Input Spaces

et al. 2020

View full text Add to dashboard Cite

During exploratory performance testing, software testers evaluate the performance of a software system with different input combinations in order to identify combinations that cause performance problems in the system under test. Performance problems such as low throughput, high response times, hangs, or crashes in software applications have an adverse effect on the customer's satisfaction. Since many of today's largescale, complex software systems (e.g., eCommerce applications, databases, web servers) exhibit very large multi-dimensional input spaces with many input parameters and large ranges, it has become costly and inefficient to explore all possible combinations of inputs in order to detect performance problems. In order to address this issue, we introduce a method for identifying input combinations that trigger performance problems in the software system under test. Our method, under the name of iPerfXRL, employs deep reinforcement learning in order to explore a given large multi-dimensional input space efficiently. The main benefit of the approach is that, during the exploration process, it learns and recognizes the problematic regions of the input space that have a higher chance of triggering performance problems. It concentrates the search in those problematic regions to find as many input combinations as possible that can trigger performance problems while executing a limited number of input combinations against the system. In addition, our approach does not require prior domain knowledge or access to the source code of the system. Therefore, it can be applied to any software system where we can interactively execute different input combinations while monitoring their performance impact on the system. We implement iPerfXRL on top of the Soft Actor-Critic algorithm. We evaluate empirically the efficiency and effectiveness of our approach against alternative state-of-the-art approaches. Our results show that iPerfXRL accurately identifies the problematic regions of the input space and finds up to 9 times more input combinations that trigger performance problems on the system under test than the alternative approaches. INDEX TERMS Exploratory performance testing, Deep reinforcement learning, Test data generation I. INTRODUCTION One of the most critical and challenging tasks for developers is to identify and fix performance problems of software systems [1]. Performance problems such as low throughput, high response times, hangs, or crashes in software applications have an adverse effect on the customer's satisfaction. According to [2], there are higher chances of a software system crashing due to performance problems rather than functional failures. Recent reports [1] show that certain input combinations can trigger more than half of the performance bottlenecks identified in non-trivial software systems. The reason is that certain input combinations can invoke inefficient code sequences or resource-intensive operations, which result in overall system performance degradation commonly referred to as performan...

show abstract

Adaptive Stress Testing for Autonomous Vehicles

Cited by 141 publications

References 16 publications

Efficient Autonomy Validation in Simulation with Adaptive Stress Testing

Efficient Autonomy Validation in Simulation with Adaptive Stress Testing

Adaptive Stress Testing with Reward Augmentation for Autonomous Vehicle Validatio

Using Deep Reinforcement Learning for Exploratory Performance Testing of Software Systems With Multi-Dimensional Input Spaces

Contact Info

Product

Resources

About