Improving fault localization for Simulink models using search-based testing and prediction models

Abstract

Abstract-One promising way to improve the accuracy of fault localization based on statistical debugging is to increase diversity among test cases in the underlying test suite. In many practical situations, adding test cases is not a cost-free option because test oracles are developed manually or running test cases is expensive. Hence, we require to have test suites that are both diverse and small to improve debugging. In this paper, we focus on improving fault localization of Simulink models by generating test cases. We identify three test objectives that aim to increase test suite diversity. We use these objectives in a search-based algorithm to generate diversified but small test suites. To further minimize test suite sizes, we develop a prediction model to stop test generation when adding test cases is unlikely to improve fault localization. We evaluate our approach using three industrial subjects. Our results show (1) the three selected test objectives are able to significantly improve the accuracy of fault localization for small test suite sizes, and (2) our prediction model is able to maintain almost the same fault localization accuracy while reducing the average number of newly generated test cases by more than half.

Effective fault localization of automotive Simulink models: achieving the trade-off between test oracle effort and fault localization accuracy

“…Our results show that using our optimal prediction model, on average, by generating only 11 test cases, we are able to obtain an accuracy improvement close to that obtained by 25 test cases when our strategy to stop test generation is not used. This paper extends a previous conference paper [43] published at the 24th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER 2017). This paper offers major extensions over our previous paper in the following areas: (1) We consider a new test objective, output diversity [53], and study its effectiveness for fault localization.…”

“…This paper offers major extensions over our previous paper in the following areas: (1) We consider a new test objective, output diversity [53], and study its effectiveness for fault localization. We further compare output diversity with the three test objectives (i.e., coverage dissimilarity, coverage density and number of dynamic basic blocks) discussed in our previous paper [43]. Our results do not indicate a significant difference in fault localization accuracy results obtained based on output diversity compared to the accuracy results obtained based on the test objectives discussed in our previous work.…”

“…As noted earlier, this article is an extension of a previous conference paper [43]. In this section, we discuss and compare with several other strands of related work in the areas of automated test generation, model debugging and predicting fault localization results.…”

“…(3) To build prediction models, we propose a new feature, RankCorrelation, defined over statistical rankings. We consider seven different feature combinations consisting of RankCorrelation and the three features (i.e., Round, SetDistance, OrderingDistance) discussed in our previous paper [43]. Among these seven feature combinations, the best performing combination is the one consisting of Round, SetDistance and OrderingDistance.…”

Automatic Failure Explanation in CPS Models

“…However, this is a black-box technique that does not attempt to open the model and explain the failure in terms of its internal signals and components. Other approaches are based on fault-localization [5,7,15,16,14], a statistical technique measuring the code coverage in the failed and successful tests. This method provides a limited explanation that does not often help the engineers to understand if the selected code is really faulty and how the fault has propagated across the components resulting on actual failure.…”