SBST Tool Competition 2021

Panichella, Sebastiano; Gambi, Alessio; Zampetti, Fiorella; Riccio, Vincenzo

doi:10.1109/sbst52555.2021.00011

Cited by 41 publications

(47 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prototype currently supports two modes of test case generation: basic, which generates reflection based test cases, and advanced, that tries to generate test cases using only public API's of the PUT. We have participated in the SBST 2021 Tool Competition [45,46] with the described prototype. With an overall score of 44.21, Kex ranked fifth.…”

Section: Evaluation Of Prototypesmentioning

confidence: 99%

Untitled

2022

ICS

View full text Add to dashboard Cite

Журнал входит в БД SCOPUS и в Перечень рецензируемых научных изданий, в которых должны быть опубликованы основные научные результаты диссертаций на соискание ученой степени кандидата наук, на соискание ученой степени доктора наук.Журнал зарегистрирован в Министерстве РФ по делам печати, телерадиовещания и средств массовых коммуникаций.

show abstract

Section: Evaluation Of Prototypesmentioning

confidence: 99%

Untitled

2022

ICS

View full text Add to dashboard Cite

show abstract

“…It is based on the DAVE-2 architecture including a normalization layer, five convolutional layers followed by three fully connected layers [34]. This test subject has been used in previous research and also in the SBST 2021 cyber-physical tool competition for evaluating test scenario generators [23], [32], [35], and moreover does not require manual training, which can mitigate the threats to the validity of the results [35]. The experiments are simulation scenarios generated by a Python test scenario generator and executed by the simulation engine.…”

Section: Empirical Evaluationmentioning

confidence: 99%

Machine Learning Testing in an ADAS Case Study Using Simulation-Integrated Bio-Inspired Search-Based Testing

Moghadam¹,

Borg²,

Saadatmand³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper presents an extended version of Deeper, a search-based simulation-integrated test solution that generates failure-revealing test scenarios for testing a deep neural network-based lane-keeping system. In the newly proposed version, we utilize a new set of bio-inspired search algorithms, genetic algorithm (GA), (µ + λ) and (µ, λ) evolution strategies (ES), and particle swarm optimization (PSO), that leverage a quality population seed and domain-specific cross-over and mutation operations tailored for the presentation model used for modeling the test scenarios. In order to demonstrate the capabilities of the new test generators within Deeper, we carry out an empirical evaluation and comparison with regard to the results of five participating tools in the cyber-physical systems testing competition at SBST 2021. Our evaluation shows the newly proposed test generators in Deeper not only represent a considerable improvement on the previous version but also prove to be effective and efficient in provoking a considerable number of diverse failure-revealing test scenarios for testing an ML-driven lane-keeping system. They can trigger several failures while promoting test scenario diversity, under a limited test time budget, high target failure severity, and strict speed limit constraints.

show abstract

“…The virtual roads are generated inside a driving simulator called BeamNG [14], which can generate photo-realistic, but synthetic, images of roads. Given such characteristics, BeamNG [14] has been also used as the main simulation platform in the 2021 edition of the SBST tool competition [53].…”

Section: Beamng and Asfaultmentioning

confidence: 99%

“…In this case, threats can be mainly due to the imprecision in simulation realism as well as the automated classification of safe and unsafe scenarios. We mitigated both threats by leveraging BeamNG (used in this year's SBST tool competition [53]) as a soft-body simulation environment (which ensures a high simulation accuracy in safety-critical scenarios) and AsFault as a reference technological solution to generate and execute test cases [30], as detailed in Section 4.…”

Section: Threats To Validitymentioning

confidence: 99%

Single and Multi-objective Test Cases Prioritization for Self-driving Cars in Virtual Environments

Birchler¹,

Khatiri²,

Derakhshanfar³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Testing with simulation environments help to identify critical failing scenarios emerging autonomous systems such as self-driving cars (SDCs) and are safer than in-field operational tests. However, these tests are very expensive and are too many to be run frequently within limited time constraints. In this paper, we investigate test case prioritization techniques to increase the ability to detect SDC regression faults with virtual tests earlier.Our approach, called SDC-Prioritizer, prioritizes virtual tests for SDCs according to static features of the roads used within the driving scenarios. These features are collected without running the tests and do not require past execution results. SDC-Prioritizer utilizes meta-heuristics to prioritize the test cases using diversity metrics (black-box heuristics) computed on these static features. Our empirical study conducted in the SDC domain shows that SDC-Prioritizer doubles the number of safety-critical failures that virtual tests can detect at the same level of execution time compared to baselines: random and greedy-based test case orderings. Furthermore, this meta-heuristic search performs statistically better than both baselines in terms of detecting safety-critical failures. SDC-Prioritizer effectively prioritize test cases for SDCs with a large improvement in fault detection while its overhead (up to 0.34% of the test execution cost) is negligible.CCS Concepts: • Software and its engineering → Search-based software engineering; Software testing and debugging.

show abstract

SBST Tool Competition 2021

Cited by 41 publications

References 18 publications

Untitled

Untitled

Machine Learning Testing in an ADAS Case Study Using Simulation-Integrated Bio-Inspired Search-Based Testing

Single and Multi-objective Test Cases Prioritization for Self-driving Cars in Virtual Environments

Contact Info

Product

Resources

About