Addressing Inherent Uncertainty: Risk-Sensitive Behavior Generation for Automated Driving using Distributional Reinforcement Learning

Bernhard, Julian; Pollok, Stefan; Knoll, Alois

doi:10.1109/ivs.2019.8813791

Cited by 27 publications

(16 citation statements)

References 22 publications

(42 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The CVaR approach does not require a backup policy and is therefore suitable for environments where such a policy is hard to define, e.g., the Atari-57 benchmark [26]. Bernhard et al first trained a risk-neutral IQN agent, then lowered the CVaR threshold after the training process had been completed, and showed a reduction of collisions in an intersection driving scenario [14]. However, such a procedure does not provide the correct estimate of the return distribution Z τ (s, a) for a risk-averse setting (α < 1) and could lead to arbitrary decisions.…”

Section: Discussionmentioning

confidence: 99%

“…However, previous studies do not estimate the aleatoric or the epistemic uncertainty of the decision that the trained agent recommends. One exception is the study by Bernhard et al, where a distributional RL approach is used to create a risk-sensitive decision-making agent [14]. However, the method is not applied in a theoretically consistent way and can therefore cause arbitrary decisions, which is further discussed in Sect.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving

Hoel¹,

Wolff²,

Laine³

2021

Preprint

View full text Add to dashboard Cite

Reinforcement learning (RL) can be used to create a decision-making agent for autonomous driving. However, previous approaches provide only black-box solutions, which do not offer information on how confident the agent is about its decisions. An estimate of both the aleatoric and epistemic uncertainty of the agent's decisions is fundamental for realworld applications of autonomous driving. Therefore, this paper introduces the Ensemble Quantile Networks (EQN) method, which combines distributional RL with an ensemble approach, to obtain a complete uncertainty estimate. The distribution over returns is estimated by learning its quantile function implicitly, which gives the aleatoric uncertainty, whereas an ensemble of agents is trained on bootstrapped data to provide a Bayesian estimation of the epistemic uncertainty. A criterion for classifying which decisions that have an unacceptable uncertainty is also introduced. The results show that the EQN method can balance risk and time efficiency in different occluded intersection scenarios, by considering the estimated aleatoric uncertainty. Furthermore, it is shown that the trained agent can use the epistemic uncertainty information to identify situations that the agent has not been trained for and thereby avoid making unfounded, potentially dangerous, decisions outside of the training distribution.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving

Hoel¹,

Wolff²,

Laine³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Existing probabilistic risk definitions often consider collision as harmful events in riskbased planning approaches [9,[17][18][19] using the state probability, the probability of spatial overlap at discrete times [20]. Value-based probabilistic risk definitions arose in finance and are applied to robotics and autonomous driving [21][22][23]. Defining risk based on collisions is infeasible considering the vast amount of samples required to approximate human collision probabilities P col ≈10 −7 .…”

Section: A Risk-metrics and Levelsmentioning

confidence: 99%

Risk-Constrained Interactive Safety under Behavior Uncertainty for Autonomous Driving

Bernhard¹,

Knoll²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Balancing safety and efficiency when planning in dense traffic is challenging. Interactive behavior planners incorporate prediction uncertainty and interactivity inherent to these traffic situations. Yet, their use of single-objective optimality impedes interpretability of the resulting safety goal. Safety envelopes which restrict the allowed planning region yield interpretable safety under the presence of behavior uncertainty, yet, they sacrifice efficiency in dense traffic due to conservative driving. Studies show that humans balance safety and efficiency in dense traffic by accepting a probabilistic risk of violating the safety envelope. In this work, we adopt this safety objective for interactive planning. Specifically, we formalize this safety objective, present the Risk-Constrained Robust Stochastic Bayesian Game modeling interactive decisions satisfying a maximum risk of violating a safety envelope under uncertainty of other traffic participants' behavior and solve it using our variant of Multi-Agent Monte Carlo Tree Search. We demonstrate in simulation that our approach outperforms baselines approaches, and by reaching the specified violation risk level over driven simulation time, provides an interpretable and tunable safety objective for interactive planning.

show abstract

“…Robustness against inaccuracies in prediction should be further evaluated, e.g. by advancing risk-aware planning [26].…”

Section: A Behavior Modelingmentioning

confidence: 99%

BARK: Open Behavior Benchmarking in Multi-Agent Environments

Bernhard

Esterle

Hart

et al. 2020

2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Self Cite

View full text Add to dashboard Cite

Predicting and planning interactive behaviors in complex traffic situations presents a challenging task. Especially in scenarios involving multiple traffic participants that interact densely, autonomous vehicles still struggle to interpret situations and to eventually achieve their own driving goal. As driving tests are costly and challenging scenarios are hard to find and reproduce, simulation is widely used to develop, test, and benchmark behavior models. However, most simulations rely on datasets and simplistic behavior models for traffic participants and do not cover the full variety of real-world, interactive human behaviors. In this work, we introduce BARK, an open-source behavior benchmarking environment designed to mitigate the shortcomings stated above. In BARK, behavior models are (re-)used for planning, prediction, and simulation. A range of models is currently available, such as Monte-Carlo Tree Search and Reinforcement Learning-based behavior models. We use a public dataset and sampling-based scenario generation to show the inter-exchangeability of behavior models in BARK. We evaluate how well the models used cope with interactions and how robust they are towards exchanging behavior models. Our evaluation shows that BARK provides a suitable framework for a systematic development of behavior models.

show abstract

Addressing Inherent Uncertainty: Risk-Sensitive Behavior Generation for Automated Driving using Distributional Reinforcement Learning

Cited by 27 publications

References 22 publications

Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving

Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving

Risk-Constrained Interactive Safety under Behavior Uncertainty for Autonomous Driving

BARK: Open Behavior Benchmarking in Multi-Agent Environments

Contact Info

Product

Resources

About