Evaluation Uncertainty in Data-Driven Self-Driving Testing

Huang, Zhiyuan; Arief, Mansur; Lam, Henry; Zhao, Ding

doi:10.1109/itsc.2019.8917406

Cited by 13 publications

(4 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…According to them, the procedure is more efficient than the random selection of test scenarios, but no quantitative statement is made. There is similar work from Huang et al [98], Huang et al [99], Huang et al [100], and Huang et al [101], [102] as well as from other members of their research group [103], [104].…”

Section: B Sampling From Parameter Distributionssupporting

confidence: 54%

Survey on Scenario-Based Safety Assessment of Automated Vehicles

et al. 2020

View full text Add to dashboard Cite

When will automated vehicles come onto the market? This question has puzzled the automotive industry and society for years. The technology and its implementation have made rapid progress over the last decade, but the challenge of how to prove the safety of these systems has not yet been solved. Since a market launch without proof of safety would neither be accepted by society nor by legislators, much time and many resources have been invested into safety assessment in recent years in order to develop new approaches for an efficient assessment. This paper therefore provides an overview of various approaches, and gives a comprehensive survey of the so-called scenario-based approach. The scenario-based approach is a promising method, in which individual traffic situations are typically tested by means of virtual simulation. Since an infinite number of different scenarios can theoretically occur in real-world traffic, even the scenario-based approach leaves the question unanswered as to how to break these down into a finite set of scenarios, and find those which are representative in order to render testing more manageable. This paper provides a comprehensive literature review of related safety-assessment publications that deal precisely with this question. Therefore, this paper develops a novel taxonomy for the scenario-based approach, and classifies all literature sources. Based on this, the existing methods will be compared with each other and, as one conclusion, the alternative concept of formal verification will be combined with the scenario-based approach. Finally, future research priorities are derived.

show abstract

Section: B Sampling From Parameter Distributionssupporting

confidence: 54%

Survey on Scenario-Based Safety Assessment of Automated Vehicles

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Some recent benchmarks [199] use realistic 3D simulators to construct real-world scenarios and use accelerated evaluation methods [10,204] to emphasize the rare safety-critical cases. However, there is a trade-off between the modeling error and evaluation error [74].…”

Section: How To Design Evaluation Platforms For Trustworthy Rl?mentioning

confidence: 99%

Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability

Xu¹,

Liu²,

Huang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

A trustworthy reinforcement learning algorithm should be competent in solving challenging real-world problems, including robustly handling uncertainties, satisfying safety constraints to avoid catastrophic failures, and generalizing to unseen scenarios during deployments. This study aims to overview these main perspectives of trustworthy reinforcement learning considering its intrinsic vulnerabilities on robustness, safety, and generalizability. In particular, we give rigorous formulations, categorize corresponding methodologies, and discuss benchmarks for each perspective. Moreover, we provide an outlook section to spur promising future directions with a brief discussion on extrinsic vulnerabilities considering human feedback. We hope this survey could bring together separate threads of studies together in a unified framework and promote the trustworthiness of reinforcement learning. CCS Concepts: • Computing methodologies → Reinforcement learning; Markov decision processes; • Security and privacy → Social aspects of security and privacy; • Computer systems organization → Robotics; • Hardware → Safety critical systems.

show abstract

“…This is of particular importance since the uncertainty in the final metrics is due to both modeling uncertainty and sampling uncertainty. While the former can be driven down by designing as accurate a model as possible, the latter can only be reduced by running longer , and larger number simulation runs [91]. A good review for this input-induced uncertainty and ways to deal with it in modeling can be found in [92]- [94].…”

Section: Metrics For Generationmentioning

confidence: 99%

A Survey on Safety-Critical Driving Scenario Generation -- A Methodological Perspective

Ding¹,

Xu²,

Arief³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Autonomous driving systems have witnessed a significant development during the past years thanks to the advance in machine learning-enabled sensing and decision-making algorithms. One critical challenge for their massive deployment in the real world is their safety evaluation. Most existing driving systems are still trained and evaluated on naturalistic scenarios collected from daily life or heuristically-generated adversarial ones. However, the large population of cars, in general, leads to an extremely low collision rate, indicating that the safety-critical scenarios are rare in the collected real-world data. Thus, methods to artificially generate scenarios become crucial to measure the risk and reduce the cost. In this survey, we focus on the algorithms of safety-critical scenario generation in autonomous driving. We first provide a comprehensive taxonomy of existing algorithms by dividing them into three categories: data-driven generation, adversarial generation, and knowledge-based generation. Then, we discuss useful tools for scenario generation, including simulation platforms and packages. Finally, we extend our discussion to five main challenges of current works -fidelity, efficiency, diversity, transferability, controllability -and research opportunities lighted up by these challenges.

show abstract

Evaluation Uncertainty in Data-Driven Self-Driving Testing

Cited by 13 publications

References 19 publications

Survey on Scenario-Based Safety Assessment of Automated Vehicles

Survey on Scenario-Based Safety Assessment of Automated Vehicles

Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability

A Survey on Safety-Critical Driving Scenario Generation -- A Methodological Perspective

Contact Info

Product

Resources

About