An Empirical Testing of Autonomous Vehicle Simulator System for Urban Driving

Seymour, J.; Ho, Dac-Thanh-Chuong; Luu, Quang-Hung

doi:10.1109/aitest52744.2021.00031

Cited by 14 publications

(12 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The key idea embedded within the framework is the use of sequential MGs in revealing undesirable behaviors of the system under test. A preliminary MT-based testing with Baidu Apollo has been conducted [36]. The extension to the modular ADS is part of our ongoing tasks for the next version of the framework.…”

Section: Discussionmentioning

confidence: 99%

A Sequential Metamorphic Testing Framework for Understanding Automated Driving Systems

Luu¹,

Liu²,

Chen³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Automated driving systems (ADS) are expected to be reliable and robust against a wide range of driving scenarios. Their decisions, first and foremost, must be well understood. Understanding a decision made by ADS is a great challenge, because it is not straightforward to tell whether the decision is correct or not, and how to verify it systematically. In this paper, a Sequential MetAmoRphic Testing (SMART) framework is proposed based on metamorphic testing, a mainstream software testing approach. In metamorphic testing, metamorphic groups are constructed by selecting multiple inputs according to the so-called metamorphic relations, which are basically the system's necessary properties; the violation of certain relations by some corresponding metamorphic groups implies the detection of erroneous system behaviors. The proposed framework makes use of sequences of metamorphic groups to understand ADS behaviors, and is applicable without the need of ground-truth datasets. To demonstrate its effectiveness, the framework is applied to test three ADS models that steer an autonomous car in different scenarios with another car either leading in front or approaching in the opposite direction. The conducted experiments reveal a large number of undesirable behaviors in these top-ranked deep learning models in the scenarios. These counter-intuitive behaviors are associated with how the core models of ADS respond to different positions, directions and properties of the other car in its proximity. Further analysis of the results helps identify critical factors affecting ADS decisions and thus demonstrates that the framework can be used to provide a comprehensive understanding of ADS before their deployment.

show abstract

Section: Discussionmentioning

confidence: 99%

A Sequential Metamorphic Testing Framework for Understanding Automated Driving Systems

Luu¹,

Liu²,

Chen³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Metamorphic relations have been discussed for the module-level testing in §6.2.2 and §6.5.2. On the system-level testing, Seymour et al [142] define a metamorphic relation that the behavior of ADS should be unchanged regardless of the weather condition, which is then applied in metamorphic testing to identify violation cases in the systems. Han et al [143] utilize metamorphic relations to distinguish between real failure and false alarms.…”

Section: Test Methodologymentioning

confidence: 99%

“…The behavior of ADS should be unchanged regardless of the weather condition [142] The behavior of the ADS should be similar in slightly different scenarios [143] Adequacy Description Literature…”

Section: Metamorphic Relationsmentioning

confidence: 99%

See 1 more Smart Citation

A Survey on Automated Driving System Testing: Landscapes and Trends

Tang¹,

Zhang²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Automated Driving Systems (ADS) have made great achievements in recent years thanks to the efforts from both academia and industry. A typical ADS is composed of multiple modules, including sensing, perception, planning and control, which brings together the latest advances in multiple domains. Despite these achievements, safety assurance of the systems is still of great significance, since the unsafe behavior of ADS can bring catastrophic consequences and unacceptable economic and social losses. Testing is an important approach to system validation for the deployment in practice; in the context of ADS, it is extremely challenging, due to the system complexity and multidisciplinarity. There has been a great deal of literature that focuses on the testing of ADS, and a number of surveys have also emerged to summarize the technical advances. However, most of these surveys focus on the system-level testing that is performed within software simulators, and thereby ignore the distinct features of individual modules. In this paper, we provide a comprehensive survey on the existing ADS testing literature, which takes into account both module-level and system-level testing. Specifically, we make the following contributions: (1) we build a threat model that reveals the potential safety threats for each module of an ADS; (2) we survey the module-level testing techniques for ADS and highlight the technical differences affected by the properties of the modules; (3) we also survey the system-level testing techniques, but we focus on empirical studies that take a bird's-eye view on the system, the problems due to the collaborations between modules, and the gaps between ADS testing in simulators and real world; (4) we identify the challenges and opportunities in ADS testing, which facilitates the future research in this field. CCS Concepts: • Computer systems organization → Embedded systems; • Software and its engineering → Software verification and validation; • Security and privacy → Systems security; • Computing methodologies → Artificial intelligence.

show abstract

“…Several studies have been conducted to use MT to test the software for scientific computations and simulations. For instances, in testing simulations, MT has been successfully adopted to test agent-based (ABM) and discrete-event (DES) simulations [26], hybrid ABM and DES systems [11], health care simulation [25], webenabled simulation [1], and simulator platform for self-driving cars [36,41]. In testing scientific software, MT is an effective technique to detect faults in simulation programs for designing nuclear power plants [14], bioinformatics programs [4], epidemiological models [31,33], chemical reaction networks for prototyping nano-scale molecular devices [13], matrix calculation programs [32], solvers for partial differential equations [3], multiple linear regression software [22], ocean modelling [16], storm water management model systems [19], machine learning-based hydro-logical models [40], Monte-Carlo computational programs [10,30], serverless scientific applications [20], as well as other types of scientific software [9,18,19,29].…”

Section: Introductionmentioning

confidence: 99%

Testing Ocean Software with Metamorphic Testing

Luu,

Liu,

Chen

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Advancing ocean science has a significant impact to the development of the world, from operating a safe navigation for vessels to maintaining a healthy and diverse ocean ecosystem. Various ocean software systems have been extensively adopted for different purposes, for instance, predicting hourly sea level elevation across shorelines, simulating large-scale ocean circulations, as well as integrating into Earth system models for weather forecasts and climate projections. Regardless of their significance, guaranteeing the trustworthiness of ocean software and modelling systems is a long-standing challenge. The testing of ocean software suffers a lot from the so-called oracle problem, which refers to the absence of test oracles mainly due to the nonlinear interactions of multiple physical variables and the high complexity in computation. In the ocean, observed tidal signals are distorted by non-deterministic physical variables, hindering us from knowing the "true" astronomical tidal constituents existing in the timeseries. In this paper, we present how to test tidal analysis and prediction (TAP) software based on metamorphic testing (MT), a simple yet effective testing approach to the oracle problem. In particular, we construct metamorphic relations from the periodic property of astronomical tide, and then use them to successfully detect a real-life defect in an open-source TAP software. We also conduct a series of experiments to further demonstrate the applicability and effectiveness of MT in the testing of TAP software. Our study not only justifies the potential of MT in testing more complex ocean software and modelling systems, but also can be expanded to assess and improve the quality of a broader range of scientific simulation software systems. CCS CONCEPTS• Applied computing → Earth and atmospheric sciences; • Software and its engineering → Software testing and debugging.

show abstract

An Empirical Testing of Autonomous Vehicle Simulator System for Urban Driving

Cited by 14 publications

References 4 publications

A Sequential Metamorphic Testing Framework for Understanding Automated Driving Systems

A Sequential Metamorphic Testing Framework for Understanding Automated Driving Systems

A Survey on Automated Driving System Testing: Landscapes and Trends

Testing Ocean Software with Metamorphic Testing

Contact Info

Product

Resources

About