ViSTA: a Framework for Virtual Scenario-based Testing of Autonomous Vehicles

Piazzoni, Andrea; Cherian, Jim; Azhar, Mohamed; Yap, Jing Yew; Shung, James Lee Wei; Vijay, Roshan

doi:10.1109/aitest52744.2021.00035

Cited by 14 publications

(17 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Each experiment is a set of simulations run identified by two elements: a perception configuration and a test case (i.e., a parametrized scenario). We implement our experiment in the ViSTA framework [22], which interfaces with SVL simulator [23] and Apollo [24] as the AV under test. We choose SVL since it facilitates the co-simulation by providing connection bridges toward independent AV stacks.…”

Section: Methodsmentioning

confidence: 99%

CoPEM: Cooperative Perception Error Models for Autonomous Driving

Piazzoni

Cherian

Vijay

et al. 2022

2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC)

Self Cite

View full text Add to dashboard Cite

In this paper, we introduce the notion of Cooperative Perception Error Models (coPEMs) towards achieving an effective and efficient integration of V2X solutions within a virtual test environment. We focus our analysis on the occlusion problem in the (onboard) perception of Autonomous Vehicles (AV), which can manifest as misdetection errors on the occluded objects. Cooperative perception (CP) solutions based on Vehicle-to-Everything (V2X) communications aim to avoid such issues by cooperatively leveraging additional points of view for the world around the AV. This approach usually requires many sensors, mainly cameras and LiDARs, to be deployed simultaneously in the environment either as part of the road infrastructure or on other traffic vehicles. However, implementing a large number of sensor models in a virtual simulation pipeline is often prohibitively computationally expensive. Therefore, in this paper, we rely on extending Perception Error Models (PEMs) to efficiently implement such cooperative perception solutions along with the errors and uncertainties associated with them. We demonstrate the approach by comparing the safety achievable by an AV challenged with a traffic scenario where occlusion is the primary cause of a potential collision.

show abstract

Section: Methodsmentioning

confidence: 99%

CoPEM: Cooperative Perception Error Models for Autonomous Driving

Piazzoni

Cherian

Vijay

et al. 2022

2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC)

Self Cite

View full text Add to dashboard Cite

show abstract

“…LGSVL [213] High-fidelity simulator supporting sensor simulation, vehicle dynamics and control simulation, etc. [21,92,125,138,142,145,[179][180][181][182]225] 2020 SCANeR Studio [248] Flexible simulator allowing simulation configuration and results analysis [104] 2020 ADAMS [215] Supporting the analysis of vehicle dynamics -2020…”

Section: 3g Videos -2020mentioning

confidence: 99%

“…LiveTCM [180] VisTA [181] Trajectory coverage [182] Route coverage [92,145] MSF attack [21] Upgraded DL Models New Data Pipeline v6.0 2020.9…”

mentioning

confidence: 99%

“…Several coverage criteria, such as route coverage [92,145] and trajectory coverage [182], are adopted for the testing of Apollo's planning algorithms. There are also testing frameworks, e.g., ViSTA [181] and LiveTCM [180], proposed to support test case generation and automated execution. These frameworks are evaluated on Apollo 5.0 and they are proved effective in exposing issues and limitations in the system.…”

mentioning

confidence: 99%

See 1 more Smart Citation

A Survey on Automated Driving System Testing: Landscapes and Trends

Tang¹,

Zhang²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Automated Driving Systems (ADS) have made great achievements in recent years thanks to the efforts from both academia and industry. A typical ADS is composed of multiple modules, including sensing, perception, planning and control, which brings together the latest advances in multiple domains. Despite these achievements, safety assurance of the systems is still of great significance, since the unsafe behavior of ADS can bring catastrophic consequences and unacceptable economic and social losses. Testing is an important approach to system validation for the deployment in practice; in the context of ADS, it is extremely challenging, due to the system complexity and multidisciplinarity. There has been a great deal of literature that focuses on the testing of ADS, and a number of surveys have also emerged to summarize the technical advances. However, most of these surveys focus on the system-level testing that is performed within software simulators, and thereby ignore the distinct features of individual modules. In this paper, we provide a comprehensive survey on the existing ADS testing literature, which takes into account both module-level and system-level testing. Specifically, we make the following contributions: (1) we build a threat model that reveals the potential safety threats for each module of an ADS; (2) we survey the module-level testing techniques for ADS and highlight the technical differences affected by the properties of the modules; (3) we also survey the system-level testing techniques, but we focus on empirical studies that take a bird's-eye view on the system, the problems due to the collaborations between modules, and the gaps between ADS testing in simulators and real world; (4) we identify the challenges and opportunities in ADS testing, which facilitates the future research in this field. CCS Concepts: • Computer systems organization → Embedded systems; • Software and its engineering → Software verification and validation; • Security and privacy → Systems security; • Computing methodologies → Artificial intelligence.

show abstract

“…To deal with such safety-related challenges, there is an increasing interest in adopting agile development paradigms within the CPS safety-critical domains [6,7] to identify hazards and elicit safety requirements iteratively [8]. Consequently, researchers proposed the usage of Digital-Twins 1 technologies to simulate and test CPSs in a diversified set of scenarios [9,10,11,12,13] to support testing automation [14,15], regression testing [12,16], and debugging [17,18] activities. In this context, simulation-based testing has been suggested as a promising direction to improve the SDC testing practices [19,20,21] because simulation environments enable efficient test execution, reproducible results, and testing under critical conditions [22].…”

Section: Introductionmentioning

confidence: 99%

Cost-effective Simulation-based Test Selection in Self-driving Cars Software with SDC-Scissor

Birchler

Ganz

Khatiri

et al. 2022

2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)

View full text Add to dashboard Cite

Simulation platforms facilitate the continuous development of complex systems such as self-driving cars (SDCs). However, previous results on testing SDCs using simulations have shown that most of the automatically generated tests do not strongly contribute to establishing confidence in the quality and reliability of the SDC. Therefore, those tests can be characterized as "uninformative", and running them generally means wasting precious computational resources. We address this issue with SDC-Scissor, a framework that leverages Machine Learning to identify simulation-based tests that are unlikely to detect faults in the SDC software under test and skip them before their execution. Consequently, by filtering out those tests, SDC-Scissor reduces the number of long-running simulations to execute and drastically increases the cost-effectiveness of simulation-based testing of SDCs software. Our evaluation concerning two large datasets and around 12'000 tests showed that SDC-Scissor achieved a higher classification F1-score (between 47% and 90%) than a randomized baseline in identifying tests that lead to a fault and reduced the time spent running uninformative tests (speedup between 107% and 170%).

show abstract

ViSTA: a Framework for Virtual Scenario-based Testing of Autonomous Vehicles

Cited by 14 publications

References 7 publications

CoPEM: Cooperative Perception Error Models for Autonomous Driving

CoPEM: Cooperative Perception Error Models for Autonomous Driving

A Survey on Automated Driving System Testing: Landscapes and Trends

Cost-effective Simulation-based Test Selection in Self-driving Cars Software with SDC-Scissor

Contact Info

Product

Resources

About