Frenetic at the SBST 2021 Tool Competition

Castellano, Ezequiel; Cetinkaya, Ahmet; Thanh, Cédric Ho; Klikovits, Stefan; Zhang, Xiaoyi; Arcaini, Paolo

doi:10.1109/sbst52555.2021.00016

Cited by 29 publications

(17 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section we compare our WOGAN algorithm to two other algorithms: random search (to establish a baseline) and the Frenetic algorithm [4], a genetic algorithm whose performance was deemed to be among the best of the SBST 2021 CPS testing competition entries [8].…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Wasserstein Generative Adversarial Networks for Online Test Generation for Cyber Physical Systems

Peltomäki¹,

Spencer²,

Porres³

2022

Preprint

View full text Add to dashboard Cite

We propose a novel online test generation algorithm WOGAN based on Wasserstein Generative Adversarial Networks. WOGAN is a general-purpose black-box test generator applicable to any system under test having a fitness function for determining failing tests. As a proof of concept, we evaluate WOGAN by generating roads such that a lane assistance system of a car fails to stay on the designated lane. We find that our algorithm has a competitive performance respect to previously published algorithms.

show abstract

Section: Resultsmentioning

confidence: 99%

“…Generating valid tests by randomly choosing sequences of plane points is difficult, so we opted to use the feature representation described in [4]. For us, a test is a vector in R 𝑑 whose components are curvature values in the range [−0.07, 0.07].…”

Section: A Novel Algorithm 21 Feature Representationmentioning

confidence: 99%

Wasserstein Generative Adversarial Networks for Online Test Generation for Cyber Physical Systems

Peltomäki¹,

Spencer²,

Porres³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…where Lev dist(i, j) indicates the weighted Levenshtein distance between the road segments. In order to provide a comparative analysis, we compare the results of the proposed test generators in Deeper with the presented test generators in the tool competition, i.e., Frenetic [25], GABExploit and GABExplore [22], Swat [26], and also the earlier version of Deeper [20]. We run the test generators on the test subject based on the same two experiment configurations as in the competition, which are shown in table I.…”

Section: Empirical Evaluationmentioning

confidence: 99%

“…In order to carry out an empirical evaluation, we use the setup provided by the cyber-physical systems (CPS) testing competition 1 We provide a comparative analysis on the performance of the proposed bio-inspired test generators in Deeper and five counterpart tools all integrated into the BeamNG simulator. In this regard, we compare the results of the proposed test generators in Deeper with five other test generator tools, Frenetic [25], GABExploit and GABExplore [22], Swat [26], and also the earlier version of the Deeper (based on NSGA-II) [20]-all participating tools in the CPS competition at SBST 2021. In order to do a fair comparison, we consider the same experimental evaluation procedures as the original CPS tool competition.…”

Section: Introductionmentioning

confidence: 99%

Machine Learning Testing in an ADAS Case Study Using Simulation-Integrated Bio-Inspired Search-Based Testing

Moghadam¹,

Borg²,

Saadatmand³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper presents an extended version of Deeper, a search-based simulation-integrated test solution that generates failure-revealing test scenarios for testing a deep neural network-based lane-keeping system. In the newly proposed version, we utilize a new set of bio-inspired search algorithms, genetic algorithm (GA), (µ + λ) and (µ, λ) evolution strategies (ES), and particle swarm optimization (PSO), that leverage a quality population seed and domain-specific cross-over and mutation operations tailored for the presentation model used for modeling the test scenarios. In order to demonstrate the capabilities of the new test generators within Deeper, we carry out an empirical evaluation and comparison with regard to the results of five participating tools in the cyber-physical systems testing competition at SBST 2021. Our evaluation shows the newly proposed test generators in Deeper not only represent a considerable improvement on the previous version but also prove to be effective and efficient in provoking a considerable number of diverse failure-revealing test scenarios for testing an ML-driven lane-keeping system. They can trigger several failures while promoting test scenario diversity, under a limited test time budget, high target failure severity, and strict speed limit constraints.

show abstract

“…Lane keeping assist system. For the lane keeping assist system, we compare AmbieGen with the open-source approach that showed the best results in the SBST2021 tool competition [34], i.e., Frenetic tool [13]. In the competition the same test evaluation pipeline was provided to all the participants.…”

Section: Experiments Designmentioning

confidence: 99%

A Search-Based Framework for Automatic Generation of Testing Environments for Cyber-Physical Systems

Humeniuk¹,

Khomh²,

Antoniol³

2022

Preprint

View full text Add to dashboard Cite

Background. Many modern cyber-physical systems incorporate computer vision technologies, complex sensors and advanced control software, allowing them to interact with the environment autonomously. Examples include drone swarms, self-driving vehicles, autonomous robots, etc. Testing such systems poses numerous challenges: not only should the system inputs be varied, but also the surrounding environment should be accounted for. A number of tools have been developed to test the system model for the possible inputs falsifying its requirements. However, they are not directly applicable to autonomous cyber-physical systems, as the inputs to their models are generated while operating in a virtual environment. Aims. In this paper, we aim to design a search-based framework, named AmbieGen, for generating diverse fault-revealing test scenarios for autonomous cyber-physical systems. The scenarios represent an environment in which an autonomous agent operates. The framework should be applicable to generating different types of environments. Method. To generate the test scenarios, we leverage the NSGA-II algorithm with two objectives. The first objective evaluates the deviation of the observed system's behaviour from its expected behaviour. The second objective is the test case diversity, calculated as a Jaccard distance with a reference test case. To guide the first objective we are using a simplified system model rather than the full model. The full model is used to run the system in the simulation environment and can take substantial time to execute (several minutes for one scenario). The simplified system model is derived from the full model and can be used to get an approximation of the results obtained from the full model without running the simulation. Results. We evaluate AmbieGen on three scenario generation case studies, namely a smart-thermostat, a robot obstacle avoidance system, and a vehicle lane-keeping assist system. For all the case studies, our approach outperforms the available baselines in fault revealing and several other metrics such as the diversity of the revealed faults and the proportion of valid test scenarios. Conclusions. AmbieGen could find scenarios, revealing failures for all the three autonomous agents considered in our case studies. We compared three configurations of AmbieGen: based on a single objective genetic algorithm, multi-objective, and random search. Both single and multi objective configurations outperform the random search. Multi objective configuration can find the individuals of the same quality as the single objective, producing more unique test scenarios in the same time budget. Our framework can be used to generate virtual environments of different types and complexity and reveal the system's faults early in the design stage.

show abstract

Frenetic at the SBST 2021 Tool Competition

Cited by 29 publications

References 2 publications

Wasserstein Generative Adversarial Networks for Online Test Generation for Cyber Physical Systems

Wasserstein Generative Adversarial Networks for Online Test Generation for Cyber Physical Systems

Machine Learning Testing in an ADAS Case Study Using Simulation-Integrated Bio-Inspired Search-Based Testing

A Search-Based Framework for Automatic Generation of Testing Environments for Cyber-Physical Systems

Contact Info

Product

Resources

About