Multi-objective black-box test case selection for cost-effectively testing simulation models

Arrieta, Aitor; Wang, Shuai; Arruabarrena, Ainhoa; Markiegi, Urtzi; Sagardui, Goiuria; Etxeberria, Leire

doi:10.1145/3205455.3205490

Cited by 29 publications

(46 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This number is similar or larger to other studies where simulation-based testing was used to evaluate testing approaches. [22][23][24] From these 99 mutants, 18 were removed from the evaluation. The reason was that the inclusion of these mutants led the system to crashing or the simulation not lasting because passengers were not attended.…”

Section: Evaluation Metricsmentioning

confidence: 99%

Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithms

Gartziandia

Arrieta

Ayerdi

et al. 2022

J Software Evolu Process

View full text Add to dashboard Cite

The software of systems of elevators needs constant maintenance to deal with new functionality, bug fixes, or legislation changes. To automatically validate the software of these systems, a typical approach in industry is to use regression oracles, which execute test inputs both in the software version under test and in a previous software version. However, these practices require a long test execution time and cannot be re‐used at different test phases. To deal with these issues, we propose Dispatching AlgoRIthm Oracle (DARIO), a test oracle that relies on regression machine‐learning algorithms to detect both functional and non‐functional problems of the system. The machine‐learning algorithms of this oracle are trained by using data from previously tested versions to predict reference functional and non‐functional performance values of the new versions. An empirical evaluation with an industrial case study demonstrates the feasibility of using our approach. A total of five regression learning algorithms were validated by using mutation testing techniques. For the context of functional bugs, the accuracy when predicting verdicts by DARIO ranged between 95% and 98%, across the different scenarios proposed. For the context of non‐functional bugs, were competitive too, having an accuracy when predicting verdicts by DARIO ranged between 83% and 87%.

show abstract

Section: Evaluation Metricsmentioning

confidence: 99%

Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithms

Gartziandia

Arrieta

Ayerdi

et al. 2022

J Software Evolu Process

View full text Add to dashboard Cite

show abstract

“…While test similarity for non-CPS systems is computed based on the lexicographic similarity for the method calls and test input, Arrieta et al measured the similarity between the test cases based on the signal values for all the states in the simulation-based test case. Test case similarity computed at the signal-level has also been investigated in the context of test case selection for CPS [9,11].…”

Section: Regression Testing For Cpssmentioning

confidence: 99%

“…In particular, we focus on prioritized simulation-based test cases to assess the lane-keeping features of self-driving cars. Instead, prior work focused on different domains, such as satellite [60], electric windows [9], industrial tanks [10,12], and cruise controller [10]. In our context, test cases consist of driving test scenarios with virtual roads (e.g., see Figure 1) and aim at assessing whether the simulated cars violate the lane-keeping requirements.…”

Section: Regression Testing For Cpssmentioning

confidence: 99%

“…Another important difference is related to the objectives (or heuristics) to optimize for regression testing. Prior work for CPS prioritize the test cases based on fault-detection capabilities [12], and diversity measured for simulation signals [9][10][11]. However, the fault-detection capability of the test cases is unknown a prior (i.e., without running the tests).…”

Section: Regression Testing For Cpssmentioning

confidence: 99%

See 1 more Smart Citation

Single and Multi-objective Test Cases Prioritization for Self-driving Cars in Virtual Environments

Birchler¹,

Khatiri²,

Derakhshanfar³

et al. 2021

Preprint

View full text Add to dashboard Cite

Testing with simulation environments help to identify critical failing scenarios emerging autonomous systems such as self-driving cars (SDCs) and are safer than in-field operational tests. However, these tests are very expensive and are too many to be run frequently within limited time constraints. In this paper, we investigate test case prioritization techniques to increase the ability to detect SDC regression faults with virtual tests earlier.Our approach, called SDC-Prioritizer, prioritizes virtual tests for SDCs according to static features of the roads used within the driving scenarios. These features are collected without running the tests and do not require past execution results. SDC-Prioritizer utilizes meta-heuristics to prioritize the test cases using diversity metrics (black-box heuristics) computed on these static features. Our empirical study conducted in the SDC domain shows that SDC-Prioritizer doubles the number of safety-critical failures that virtual tests can detect at the same level of execution time compared to baselines: random and greedy-based test case orderings. Furthermore, this meta-heuristic search performs statistically better than both baselines in terms of detecting safety-critical failures. SDC-Prioritizer effectively prioritize test cases for SDCs with a large improvement in fault detection while its overhead (up to 0.34% of the test execution cost) is negligible.CCS Concepts: • Software and its engineering → Search-based software engineering; Software testing and debugging.

show abstract

“…These models support engineers in a number of activities, most notably in automated code generation and early testing and simulation of CPS. Recent technological advancements in the areas of robotics and autonomous systems have led to increasingly more complex CPS whose models are often characterized as compute-intensive [20,22,29,53,85]. Compute-Intensive CPS models (CI-CPS) require a lot of computational power to execute [21] since they include complex computations such as dynamic, nonlinear and non-algebraic mathematics, and further, they have to be executed for long durations in order to thoroughly exercise interactions between the CPS and its environment.…”

Section: Introductionmentioning

confidence: 99%

Approximation-Refinement Testing of Compute-Intensive Cyber-Physical Models: An Approach Based on System Identification

Menghi,

Nejati,

Briand

et al. 2019

Preprint

View full text Add to dashboard Cite

Black-box testing has been extensively applied to test models of Cyber-Physical systems (CPS) since these models are not often amenable to static and symbolic testing and verification. Black-box testing, however, requires to execute the model under test for a large number of candidate test inputs. This poses a challenge for a large and practically-important category of CPS models, known as compute-intensive CPS (CI-CPS) models, where a single simulation may take hours to complete. We propose a novel approach, namely ARIsTEO, to enable effective and efficient testing of CI-CPS models. Our approach embeds black-box testing into an iterative approximation-refinement loop. At the start, some sampled inputs and outputs of the CI-CPS model under test are used to generate a surrogate model that is faster to execute and can be subjected to black-box testing. Any failure-revealing test identified for the surrogate model is checked on the original model. If spurious, the test results are used to refine the surrogate model to be tested again. Otherwise, the test reveals a valid failure. We evaluated ARIsTEO by comparing it with S-Taliro, an open-source and industry-strength tool for testing CPS models. Our results, obtained based on five publicly-available CPS models, show that, on average, ARIsTEO is able to find 24% more requirements violations than S-Taliro and is 31% faster than S-Taliro in finding those violations. We further assessed the effectiveness and efficiency of ARIsTEO on a large industrial case study from the satellite domain. In contrast to S-Taliro, ARIsTEO successfully tested two different versions of this model and could identify three requirements violations, requiring four hours, on average, for each violation.1 Machine M1: 12-core Intel Core i7 3.20GHz 32GB of RAM.

show abstract

Multi-objective black-box test case selection for cost-effectively testing simulation models

Cited by 29 publications

References 23 publications

Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithms

Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithms

Single and Multi-objective Test Cases Prioritization for Self-driving Cars in Virtual Environments

Approximation-Refinement Testing of Compute-Intensive Cyber-Physical Models: An Approach Based on System Identification

Contact Info

Product

Resources

About