How high will it be? Using machine learning models to predict branch coverage in automated testing

Grano, Giovanni; Titov, Timofey V.; Panichella, Sebastiano; Gall, Harald C.

doi:10.1109/maltesque.2018.8368454

Cited by 25 publications

(19 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, other Machine Learning applications have been experimented for software testing. Daka et al [22] adopted the readability prediction model originally proposed by Buse and Weimer [23] in the context of automatic test case generation with the goal of improving the comprehensibility of the generated tests, while Grano et al [24] preliminarily assessed the feasibility of branch coverage prediction models, showing promising results. Our work can be seen as complementary with respect to these papers, as it aims at estimating test-case effectiveness as measured by mutation score.…”

Section: Related Workmentioning

confidence: 99%

Lightweight Assessment of Test-Case Effectiveness Using Source-Code-Quality Indicators

Grano

Palomba

Gall

2021

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

Test cases are crucial to help developers preventing the introduction of software faults. Unfortunately, not all the tests are properly designed or can effectively capture faults in production code. Some measures have been defined to assess test-case effectiveness: the most relevant one is the mutation score, which highlights the quality of a test by generating the so-called mutants, ie variations of the production code that make it faulty and that the test is supposed to identify. However, previous studies revealed that mutation analysis is extremely costly and hard to use in practice. The approaches proposed by researchers so far have not been able to provide practical gains in terms of mutation testing efficiency. This leaves the problem of efficiently assessing test-case effectiveness as still open. In this paper, we investigate a novel, orthogonal, and lightweight methodology to assess test-case effectiveness: in particular, we study the feasibility to exploit production and test-code-quality indicators to estimate the mutation score of a test case. We firstly select a set of 67 factors and study their relation with test-case effectiveness. Then, we devise a mutation score estimation model exploiting such factors and investigate its performance as well as its most relevant features. The key results of the study reveal that our estimation model only based on static features has 86% of both F-Measure and AUC-ROC. This means that we can estimate the test-case effectiveness, using source-code-quality indicators, with high accuracy and without executing the tests. As a consequence, we can provide a practical approach that is beyond the typical limitations of current mutation testing techniques.Abstract-Test cases are crucial to help developers preventing the introduction of software faults. Unfortunately, not all the tests are properly designed or can effectively capture faults in production code. Some measures have been defined to assess test-case effectiveness: the most relevant one is the mutation score, which highlights the quality of a test by generating the so-called mutants, i.e., variations of the production code that make it faulty and that the test is supposed to identify. However, previous studies revealed that mutation analysis is extremely costly and hard to use in practice. The approaches proposed by researchers so far have not been able to provide practical gains in terms of mutation testing efficiency. This leaves the problem of efficiently assessing test-case effectiveness as still open. In this paper, we investigate a novel, orthogonal, and lightweight methodology to assess test-case effectiveness: in particular, we study the feasibility to exploit production and test-code-quality indicators to estimate the mutation score of a test case. We firstly select a set of 67 factors and study their relation with test-case effectiveness. Then, we devise a mutation score estimation model exploiting such factors and investigate its performance as well as its most relevant features. The key results of the s...

show abstract

Section: Related Workmentioning

confidence: 99%

Lightweight Assessment of Test-Case Effectiveness Using Source-Code-Quality Indicators

Grano

Palomba

Gall

2021

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

show abstract

“…At the end of such a procedure, the RANDOM FOREST REGRESSOR results the best algorithm for our prediction model: Indeed, it achieves the best MAE for all the possible models built experimenting the two tools and the four search budget. Is is worth to note that is the first improvement over our previous work . In fact, in this study, we introduce the RFC, comparing it with the three algorithms previously evaluated, ie, Huber Regression, Support Vector Regression, and Multilayer Perceptron.…”

Section: Resultsmentioning

confidence: 99%

“…We averaged the results of different generations because of the nondeterministic nature of the algorithms underlying test‐data generation tools. It is worth to note that such a multiple execution represents an improvement over our previous work, where we executed the tools once per class and with the default search budget only.…”

Section: Data Set and Features Descriptionmentioning

confidence: 99%

“…To capture even more fine‐grained details, we include the counts for 52 Java keywords, including keywords such as synchronized, import, or instanceof. In addition to the ones used in our preliminary study, we also include the Halstead metrics . These metrics aim at measuring the complexity of a given program looking at its operators and operands.…”

Section: Data Set and Features Descriptionmentioning

confidence: 99%

“…This paper is an extension of our seminal work, in which we firstly propose to predict the branch coverage that will be achieved by test‐data generation tools. Following, we summarize the novel contribution of this paper in respect to the original one: We introduce a new set of features, ie, the Halstead metrics .…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Branch coverage prediction in automated testing

Grano

Titov

Panichella

et al. 2019

J Software Evolu Process

Self Cite

View full text Add to dashboard Cite

Software testing is crucial in continuous integration (CI). Ideally, at every commit, all the test cases should be executed, and moreover, new test cases should be generated for the new source code. This is especially true in a Continuous Test Generation (CTG) environment, where the automatic generation of test cases is integrated into the continuous integration pipeline. In this context, developers want to achieve a certain minimum level of coverage for every software build. However, executing all the test cases and, moreover, generating new ones for all the classes at every commit is not feasible. As a consequence, developers have to select which subset of classes has to be tested and/or targeted by test-case generation. We argue that knowing a priori the branch coverage that can be achieved with test-data generation tools can help developers into taking informed decision about those issues. In this paper, we investigate the possibility to use source-code metrics to predict the coverage achieved by test-data generation tools. We use four different categories of source-code features and assess the prediction on a large data set involving more than 3'000 Java classes. We compare different machine learning algorithms and conduct a fine-grained feature analysis aimed at investigating the factors that most impact the prediction accuracy. Moreover, we extend our investigation to four different search budgets. Our evaluation shows that the best model achieves an average 0.15 and 0.21 MAE on nested cross-validation over the different budgets, respectively, on EVOSUITE and RANDOOP. Finally, the discussion of the results demonstrate the relevance of coupling-related features for the prediction accuracy. KEYWORDS automated software testing, coverage prediction, machine learning, software testing INTRODUCTIONSoftware testing is widely recognized as a crucial task in any software development process, 1 estimated at being at least about half of the entire development cost. 2,3 In the last years, we witnessed a wide adoption of continuous integration (CI) practices, where new or changed code is integrated extremely frequently into the main codebase. Testing plays an important role in such a pipeline: In an ideal world, at every single commit, every system's test case should be executed (regression testing). Moreover, additional test cases might be automatically generated to test all the new -or modified-code introduced into the main codebase. 4 This is especially true in a Continuous Test Generation (CTG) environment, where the generation of test cases is directly integrated into the continuous integration cycle. 4 However, because of the time constraints between frequent commits, a complete regression testing is not feasible for large projects. 5 Furthermore, even test suite augmentation, 6 ie, the automatic generation considering code changes and their effect on the previous codebase, is hardly doable because of the extensive amount of time needed to generate tests for just a single class. As developers want to ensure a cer...

show abstract

Towards automatically identifying the co‐change of production and test code

Huang,

Tang,

Chen

et al. 2024

Software Testing Verif & Rel

View full text Add to dashboard Cite

In software evolution, keeping the test code co‐change with the production code is important, because the outdated test code may not work and is ineffective in revealing faults in the production code. However, due to the tight development time, the production and test code may not be co‐changed immediately by developers. For example, we analysed the top 1003 popular Java projects on GitHub and found that nearly 9.3% of cases (i.e., 464,417) did not update their production and test code at the same time, that is, the production code is updated first, and then the test code is updated at intervals. The result indicates that much test code will not be updated in time. In this paper, we propose a novel approach, Jtup, to remind developers to co‐change the production code and test code in time. Specifically, we first define the co‐changed production and test code as a positive instance, while unchanged test code (i.e., production code changed and test code unchanged) as a negative instance. Then, we extract multidimensional features from the production code to characterize the possibility of their co‐change, including code change features, code complexity features, and code semantic features. Finally, several machine learning‐based methods are employed to identify the co‐changed production and test code. We conduct comprehensive experiments on 20 datasets, and the results show that the Accuracy, Precision, and Recall achieved by Jtup are 76.7%, 78.1%, and 77.4%, which outperforms the state‐of‐the‐art method.

show abstract

How high will it be? Using machine learning models to predict branch coverage in automated testing

Cited by 25 publications

References 26 publications

Lightweight Assessment of Test-Case Effectiveness Using Source-Code-Quality Indicators

Lightweight Assessment of Test-Case Effectiveness Using Source-Code-Quality Indicators

Branch coverage prediction in automated testing

Towards automatically identifying the co‐change of production and test code

Contact Info

Product

Resources

About