Digital Twins Are Not Monozygotic – Cross-Replicating ADAS Testing in Two Industry-Grade Automotive Simulators

Borg, Markus; Abdessalem, Raja Ben; Nejati, Shiva; Jegeden, Francois-Xavier; Shin, Donghwan

doi:10.1109/icst49551.2021.00050

Cited by 21 publications

(12 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As previously mentioned, digital twins have supported several testing activities, at the different X-in-the-Loop test level, including test generation [4,5,6] or anomalies detection [9]. Our hypothesis has as presumes that multi-fidelity digital twins can support on significantly enhancing several test, verification and validation activities (e.g., test generation, test oracle generation, debugging) that are not supported with common digital twins.…”

Section: Multi-fidelity Digital Twinsmentioning

confidence: 89%

“…Furthermore, digital twins highly rely on their simulation environment (e.g., Simulink) and solver-specific configurations. For instance, a recent study showed that the same digital twin can produce different results for the same test scenario when being executed in two different environments [4]. In a similar line, if the digital twin is to be used locally, in parallel with the real execution of the CPS, this shall not be a high priority task in order not to interfere safety-critical functionalities.…”

Section: Multi-fidelity Digital Twinsmentioning

confidence: 99%

“…Secondly, tests need to be executed at different test levels (i.e., Model, Software and Hardware-in-the-Loop test level). In addition, a recent study also recommended executing the same test scenarios at the same test levels but combining different simulators as the test results might differ [4], which would, again, increase the overall test execution time. Thirdly, as different engineering domains exists when developing CPSs, simulation is often turned into a co-simulation, which increments the execution time of test cases due to the need of intercommunicating between simulation tools.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Multi-Fidelity Digital Twins: a Means for Better Cyber-Physical Systems Testing?

Arrieta

2021

Preprint

View full text Add to dashboard Cite

Context: Cyber-Physical Systems (CPSs) combine software and physical components. These systems are widely applied in society within many domains, including the automotive, aerospace, railway, etc. Testing these systems is extremely challenging, therefore, it has attracted significant attention from the research community. A driving CPS testing technique in industry is simulation-based testing. However, this poses significant challenges.Objectives: To propose a novel approach that deals with the CPS testing problems.Method: In this new-idea paper we present the concept of multi-fidelity digital twins to deal with CPSs testing problems.Results: An initial hypothesis and open research questions are presented. Conclusion: The paper presents the research foundation on the concept of multi-fidelity digital twins and provides a set of unexplored research areas.

show abstract

Section: Multi-fidelity Digital Twinsmentioning

confidence: 89%

Section: Multi-fidelity Digital Twinsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Fidelity Digital Twins: a Means for Better Cyber-Physical Systems Testing?

Arrieta

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Our current study responds to this call, and motivates our work on systems testing in simulated environments. With regard to a different perspective, Borg et al [50] discuss the consistency between the test results obtained from running the same experiments based on two different simulators and investigate the reproducibility of the results in both simulators. When running the same testing campaign in PreScan and ESI Pro-SiVIC, the authors found notable differences in the test outputs related to revealed safety violations and the dynamics of cars and pedestrians.…”

Section: Related Workmentioning

confidence: 99%

Machine Learning Testing in an ADAS Case Study Using Simulation-Integrated Bio-Inspired Search-Based Testing

Moghadam¹,

Borg²,

Saadatmand³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

This paper presents an extended version of Deeper, a search-based simulation-integrated test solution that generates failure-revealing test scenarios for testing a deep neural network-based lane-keeping system. In the newly proposed version, we utilize a new set of bio-inspired search algorithms, genetic algorithm (GA), (µ + λ) and (µ, λ) evolution strategies (ES), and particle swarm optimization (PSO), that leverage a quality population seed and domain-specific cross-over and mutation operations tailored for the presentation model used for modeling the test scenarios. In order to demonstrate the capabilities of the new test generators within Deeper, we carry out an empirical evaluation and comparison with regard to the results of five participating tools in the cyber-physical systems testing competition at SBST 2021. Our evaluation shows the newly proposed test generators in Deeper not only represent a considerable improvement on the previous version but also prove to be effective and efficient in provoking a considerable number of diverse failure-revealing test scenarios for testing an ML-driven lane-keeping system. They can trigger several failures while promoting test scenario diversity, under a limited test time budget, high target failure severity, and strict speed limit constraints.

show abstract

“…In [24] a comparison between testing of DNN-based ADAS using real-world and simulator-generated data is conducted and it is also showed that how on-line and off-line testing of these systems can differ and meanwhile complement each other. Markus et al studied the consistency between the results obtained from two different simulators and investigated whether the obtained results could be mutually reproducible in both simulators [13].…”

Section: Related Workmentioning

confidence: 99%

Efficient and Effective Generation of Test Cases for Pedestrian Detection -- Search-based Software Testing of Baidu Apollo in SVL

Ebadi¹,

Moghadam²,

Borg³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

With the growing capabilities of autonomous vehicles, there is a higher demand for sophisticated and pragmatic quality assurance approaches for machine learning-enabled systems in the automotive AI context. The use of simulation-based prototyping platforms provides the possibility for early-stage testing, enabling inexpensive testing and the ability to capture critical corner-case test scenarios. Simulation-based testing properly complements conventional on-road testing. However, due to the large space of test input parameters in these systems, the efficient generation of effective test scenarios leading to the unveiling of failures is a challenge.This paper presents a study on testing pedestrian detection and emergency braking system of the Baidu Apollo autonomous driving platform within the SVL simulator. We propose an evolutionary automated test generation technique that generates failure-revealing scenarios for Apollo in the SVL environment. Our approach models the input space using a generic and flexible data structure and benefits a multi-criteria safety-based heuristic for the objective function targeted for optimization. This paper presents the results of our proposed test generation technique in the 2021 IEEE Autonomous Driving AI Test Challenge. In order to demonstrate the efficiency and effectiveness of our approach, we also report the results from a baseline random generation technique. Our evaluation shows that the proposed evolutionary test case generator is more effective at generating failure-revealing test cases and provides higher diversity between the generated failures than the random baseline.

show abstract

Digital Twins Are Not Monozygotic – Cross-Replicating ADAS Testing in Two Industry-Grade Automotive Simulators

Abstract: This is a repository copy of Digital Twins Are Not Monozygotic -cross-replicating ADAS testing in two industry-grade automotive simulators.

Cited by 21 publications

References 20 publications

Multi-Fidelity Digital Twins: a Means for Better Cyber-Physical Systems Testing?

Multi-Fidelity Digital Twins: a Means for Better Cyber-Physical Systems Testing?

Machine Learning Testing in an ADAS Case Study Using Simulation-Integrated Bio-Inspired Search-Based Testing

Efficient and Effective Generation of Test Cases for Pedestrian Detection -- Search-based Software Testing of Baidu Apollo in SVL

Contact Info

Product

Resources

About