Estimating software testing complexity

Ferrer, Javier; Chicano, Francisco; Alba, Enrique

doi:10.1016/j.infsof.2013.07.007

Cited by 29 publications

(15 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The closer work to what we present in this paper is the one of Ferrer et al [15]. They proposed the Branch Coverage Expectation (BCE) metric as the difficulty for a computer to generate test cases.…”

Section: Related Workmentioning

confidence: 81%

How high will it be? Using machine learning models to predict branch coverage in automated testing

Grano

Titov

Panichella

et al. 2018

2018 IEEE Workshop on Machine Learning Techniques for Software Quality Evaluation (MaLTeSQuE)

View full text Add to dashboard Cite

Software testing is a crucial component in modern continuous integration development environment. Ideally, at every commit, all the system's test cases should be executed and moreover, new test cases should be generated for the new code. This is especially true in the a Continuous Test Generation (CTG) environment, where the automatic generation of test cases is integrated into the continuous integration pipeline. Furthermore, developers want to achieve a minimum level of coverage for every build of their systems. Since both executing all the test cases and generating new ones for all the classes at every commit is not feasible, they have to select which subset of classes has to be tested. In this context, knowing a priori the branch coverage that can be achieved with test data generation tools might gives some useful indications for answering such a question. In this paper, we take the first steps towards the definition of machine learning models to predict the branch coverage achieved by test data generation tools. We conduct a preliminary study considering well known code metrics as a features. Despite the simplicity of these features, our results show that using machine learning to predict branch coverage in automated testing is a viable and feasible option.

show abstract

Section: Related Workmentioning

confidence: 81%

How high will it be? Using machine learning models to predict branch coverage in automated testing

Grano

Titov

Panichella

et al. 2018

2018 IEEE Workshop on Machine Learning Techniques for Software Quality Evaluation (MaLTeSQuE)

View full text Add to dashboard Cite

show abstract

“…Branch Coverage Expectation (BCE) is a complexity measure introduced by Ferrer et al in 2013 [18]. Their research has evaluated BCE with the existing complexity measures and have theoretically proved that BCE is a promising way of measuring complexity.…”

Section: A Related Workmentioning

confidence: 99%

Test Case Prioritization Approach for Sequence of Events Using Complexity Factor

Ja’afar¹,

Hassan²,

Baharom³

et al. 2021

Int. J. Adv. Sci. Eng. Inf. Technol.

View full text Add to dashboard Cite

Test case prioritization (TCP) is a method to prioritize and schedule test cases. Some approaches have been introduced to minimize the time, cost, and effort for testing the software based on the test cases that are higher priority. Since the more complex the software program, the more intensive the test should be carried out. Thus, complexity is one of the factors that affect the effectiveness of the test case prioritization. However, the existing approaches for measuring complexity have some limitations. This is due to inaccuracy in finding the weightage value for complexity as the value is useful to determine the test case prioritization. Consequently, a complexity metric measurement is needed to determine the weightage value. Hence, this paper presents work on TCP using complexity factors to enhance the accuracy in prioritizing the test cases for event sequences. This work uses Branch Coverage Expectation (BCE) for complexity measurement, in which BCE has been proven its usefulness empirically in the previous research. The event-weightage value based on the complexity is then assigned and used to prioritize the test cases while the Average Percentage of Fault Detected (APFD) metric is used to evaluate the proposed approach. A tool has been developed to ease the process as well as to facilitate the evaluation purposes. The results show the need to combine the complexity factor with other factors to improve the proposed TCP's effectiveness.

show abstract

“…The closer work to what we present in this paper is the one of Ferrer et al They proposed the Branch Coverage Expectation (BCE) metric as the difficulty for a computer to generate test cases. The definition of such a metric is based on a Markov model of the program.…”

Section: Related Workmentioning

confidence: 99%

Branch coverage prediction in automated testing

Grano

Titov

Panichella

et al. 2019

J Software Evolu Process

View full text Add to dashboard Cite

Software testing is crucial in continuous integration (CI). Ideally, at every commit, all the test cases should be executed, and moreover, new test cases should be generated for the new source code. This is especially true in a Continuous Test Generation (CTG) environment, where the automatic generation of test cases is integrated into the continuous integration pipeline. In this context, developers want to achieve a certain minimum level of coverage for every software build. However, executing all the test cases and, moreover, generating new ones for all the classes at every commit is not feasible. As a consequence, developers have to select which subset of classes has to be tested and/or targeted by test-case generation. We argue that knowing a priori the branch coverage that can be achieved with test-data generation tools can help developers into taking informed decision about those issues. In this paper, we investigate the possibility to use source-code metrics to predict the coverage achieved by test-data generation tools. We use four different categories of source-code features and assess the prediction on a large data set involving more than 3'000 Java classes. We compare different machine learning algorithms and conduct a fine-grained feature analysis aimed at investigating the factors that most impact the prediction accuracy. Moreover, we extend our investigation to four different search budgets. Our evaluation shows that the best model achieves an average 0.15 and 0.21 MAE on nested cross-validation over the different budgets, respectively, on EVOSUITE and RANDOOP. Finally, the discussion of the results demonstrate the relevance of coupling-related features for the prediction accuracy. KEYWORDS automated software testing, coverage prediction, machine learning, software testing INTRODUCTIONSoftware testing is widely recognized as a crucial task in any software development process, 1 estimated at being at least about half of the entire development cost. 2,3 In the last years, we witnessed a wide adoption of continuous integration (CI) practices, where new or changed code is integrated extremely frequently into the main codebase. Testing plays an important role in such a pipeline: In an ideal world, at every single commit, every system's test case should be executed (regression testing). Moreover, additional test cases might be automatically generated to test all the new -or modified-code introduced into the main codebase. 4 This is especially true in a Continuous Test Generation (CTG) environment, where the generation of test cases is directly integrated into the continuous integration cycle. 4 However, because of the time constraints between frequent commits, a complete regression testing is not feasible for large projects. 5 Furthermore, even test suite augmentation, 6 ie, the automatic generation considering code changes and their effect on the previous codebase, is hardly doable because of the extensive amount of time needed to generate tests for just a single class. As developers want to ensure a cer...

show abstract

Estimating software testing complexity

Cited by 29 publications

References 25 publications

How high will it be? Using machine learning models to predict branch coverage in automated testing

How high will it be? Using machine learning models to predict branch coverage in automated testing

Test Case Prioritization Approach for Sequence of Events Using Complexity Factor

Branch coverage prediction in automated testing

Contact Info

Product

Resources

About