Black-Box Test Generation from Inferred Models

Papadopoulos, Petros; Walkinshaw, Neil

doi:10.1109/raise.2015.11

Cited by 18 publications

(14 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Program comprehension is perhaps the most well-known application of specification mining [4], but the extent of its applications goes far beyond program comprehension. A great deal of research has been done to use mined specification models for test generation [5] [6] [7] [8]. Requirements engineering is another application of specification mining [9].…”

Section: Background and Related Workmentioning

confidence: 99%

An Empirical Study on Practicality of Specification Mining Algorithms on a Real-World Application

Mashhadi

Hemmati

2019

2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC)

View full text Add to dashboard Cite

Dynamic model inference techniques have been the center of many research projects recently. There are now multiple open source implementations of state-of-the-art algorithms, which provide basic abstraction and merging capabilities. Most of these tools and algorithms have been developed with one particular application in mind, which is program comprehension. The output models can abstract away the details of the program and represent the software behaviour in a concise and easy to understand form. However, one application context that is less studied is using such inferred models for debugging, where the behaviour to abstract is a faulty behaviour (e.g., a set of execution traces including a failed test case). We tried to apply some of the existing model inference techniques in a real-world industrial context to support program comprehension for debugging. Our initial experiments have shown many limitations both in terms of implementation as well as the algorithms. The paper will discuss the root cause of the failures and proposes ideas for future improvement.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

An Empirical Study on Practicality of Specification Mining Algorithms on a Real-World Application

Mashhadi

Hemmati

2019

2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC)

View full text Add to dashboard Cite

show abstract

“…PAC provides a theoretical framework for evaluating model accuracy. The same method of learning a decision tree from a test suite using the C4.5 algorithm is applied by Papadopoulos and Walkinshaw in [11]. In [11] the authors use the decision tree as input model for test case generation to extend an existing test suite.…”

Section: Related Workmentioning

confidence: 99%

“…The same method of learning a decision tree from a test suite using the C4.5 algorithm is applied by Papadopoulos and Walkinshaw in [11]. In [11] the authors use the decision tree as input model for test case generation to extend an existing test suite. In [5] the authors demonstrate empirically that there is a low to moderate correlation between coverage and effectiveness of a test suite.…”

Section: Related Workmentioning

confidence: 99%

Empirical study of correlation between mutation score and model inference based test suite adequacy assessment

Felbinger

Wotawa

Nica

2016

Proceedings of the 11th International Workshop on Automation of Software Test

View full text Add to dashboard Cite

In this paper we investigate a method for test suite evaluation that is based on an inferred model from the test suite. The idea is to use the similarity between the inferred model and the system under test as a measure of test suite adequacy, which is the ability of a test suite to expose errors in the system under test. We define similarity using the root mean squared error computed from the differences of the system under test output and the model output for certain inputs not used for model inference. In the paper we introduce the approach and provide results of an experimental evaluation where we compare the similarity with the mutation score. We used the Pearson Correlation coefficient to calculate whether a linear correlation between mutation score and root mean squared error exists. As a result we obtain that in certain cases the computed similarity strongly correlates with the mutation score.

show abstract

“…Behavioral models (e.g., state machines) are typically inferred from a running system by abstracting the execution traces. The inferred models are useful artifacts in many use cases where the actual behavior (abstracted as the inferred model) of the system is needed for analysis, such as debugging [2,51,68], testing [18,57,66,75], anomalous behavior detection [74], and requirements engineering [20].…”

Section: Introductionmentioning

confidence: 99%

Hybrid deep neural networks to infer state models of black-box systems

Mashhadi

Hemmati

2020

Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

View full text Add to dashboard Cite

Inferring behavior model of a running software system is quite useful for several automated software engineering tasks, such as program comprehension, anomaly detection, and testing. Most existing dynamic model inference techniques are white-box, i.e., they require source code to be instrumented to get run-time traces. However, in many systems, instrumenting the entire source code is not possible (e.g., when using black-box third-party libraries) or might be very costly. Unfortunately, most black-box techniques that detect states over time are either univariate, or make assumptions on the data distribution, or have limited power for learning over a long period of past behavior. To overcome the above issues, in this paper, we propose a hybrid deep neural network that accepts as input a set of time series, one per input/output signal of the system, and applies a set of convolutional and recurrent layers to learn the non-linear correlations between signals and the patterns, over time. We have applied our approach on a real UAV auto-pilot solution from our industry partner with half a million lines of C code. We ran 888 random recent system-level test cases and inferred states, over time. Our comparison with several traditional time series change point detection techniques showed that our approach improves their performance by up to 102%, in terms of finding state change points, measured by F1 score. We also showed that our state classification algorithm provides on average 90.45% F1 score, which improves traditional classification algorithms by up to 17%. CCS CONCEPTS• Computing methodologies → Neural networks; • Software and its engineering → Software reverse engineering; Requirements analysis.

show abstract

Black-Box Test Generation from Inferred Models

Cited by 18 publications

References 23 publications

An Empirical Study on Practicality of Specification Mining Algorithms on a Real-World Application

An Empirical Study on Practicality of Specification Mining Algorithms on a Real-World Application

Empirical study of correlation between mutation score and model inference based test suite adequacy assessment

Hybrid deep neural networks to infer state models of black-box systems

Contact Info

Product

Resources

About