Choosing the fitness function for the job: Automated generation of test suites that detect real faults

Salahirad, Alireza; Almulla, Hussein

doi:10.1002/stvr.1701

Cited by 31 publications

(24 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finding new exceptions requires blind guessing. Past research showed that general functions such as Branch Coverage were able to trigger more exceptions simply by guiding the search to visit more of the codebase [73].…”

Section: Technical Approach and Implementationmentioning

confidence: 99%

Learning How to Search: Generating Exception-Triggering Tests Through Adaptive Fitness Function Selection

Almulla

2020

2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)

Self Cite

View full text Add to dashboard Cite

Search-based test generation is guided by feedback from one or more fitness functions-scoring functions that judge solution optimality. Choosing informative fitness functions is crucial to meeting the goals of a tester. Unfortunately, many goals-such as forcing the class-under-test to throw exceptions, increasing test suite diversity, and attaining Strong Mutation Coverage-do not have effective fitness function formulations. We propose that meeting such goals requires treating fitness function identification as a secondary optimization step. An adaptive algorithm that can vary the selection of fitness functions could adjust its selection throughout the generation process to maximize goal attainment, based on the current population of test suites. To test this hypothesis, we have implemented two reinforcement learning algorithms in the EvoSuite unit test generation framework, and used these algorithms to dynamically set the fitness functions used during generation for the three goals identified above. We have evaluated our framework, EvoSuiteFIT, on a set of real Java case examples. EvoSuiteFIT techniques attain significant improvements for two of the three goals, and show small improvements on the third when the number of generations of evolution is fixed. Additionally, for all goals, EvoSuiteFIT detects faults missed by the other techniques. The ability to adjust fitness functions allows Evo-SuiteFIT to make strategic choices that efficiently produce more effective test suites, and examining its choices offers insight into how to attain our testing goals. We find that AFFS is a powerful technique to apply when an effective fitness function does not already exist for generating tests to achieve a testing goal.

show abstract

Section: Technical Approach and Implementationmentioning

confidence: 99%

Learning How to Search: Generating Exception-Triggering Tests Through Adaptive Fitness Function Selection

Almulla

2020

2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Dynamic method generates test data based on actual operation of the program, and the process is determinate [15]. This kind of methods need quite long time to generate test data, and is very sensitive to the initial test data [16]- [18].…”

Section: Related Workmentioning

confidence: 99%

Testing Method for Software With Randomness Using Genetic Algorithm

Yao

Gong

et al. 2020

IEEE Access

View full text Add to dashboard Cite

The main task of software testing is the generation of test data with high quality in restricted time. But for software with uncertainties, such as randomness, existing methods of generating test data often lose their effectiveness. For software with randomness, a test datum might cover the test target in one run, but obtains different results in the next run. Therefore, special testing methods must be exploited for this kind of software. In order to test software with randomness, we first propose a novel test adequacy criterion, and then build a mathematical model for generating test data according to the new criterion. Finally, we present a method of solving the optimization model using a set-based genetic algorithm. We apply the proposed method to test 12 programs, and compare with traditional genetic algorithm and the random method. From the experimental results we can see that, the proposed adequacy criterion is available for the software with randomness and the proposed algorithm can effectively generate test data.

show abstract

“…This search can then be automated. Given a measurable goal, a metaheuristic optimization algorithm can systematically sample the space of possible test input and manipulate those samples, guided by feedback from one or more fitness functionsscoring functions that judge the optimality of the chosen input (Salahirad et al 2019). In other words: algorithm + fitness functions =⇒ goal.…”

Section: Introductionmentioning

confidence: 99%

Learning how to search: generating effective test cases through adaptive fitness function selection

Almulla

2022

Empir Software Eng

Self Cite

View full text Add to dashboard Cite

Search-based test generation is guided by feedback from one or more fitness functions—scoring functions that judge solution optimality. Choosing informative fitness functions is crucial to meeting the goals of a tester. Unfortunately, many goals—such as forcing the class-under-test to throw exceptions, increasing test suite diversity, and attaining Strong Mutation Coverage—do not have effective fitness function formulations. We propose that meeting such goals requires treating fitness function identification as a secondary optimization step. An adaptive algorithm that can vary the selection of fitness functions could adjust its selection throughout the generation process to maximize goal attainment, based on the current population of test suites. To test this hypothesis, we have implemented two reinforcement learning algorithms in the EvoSuite unit test generation framework, and used these algorithms to dynamically set the fitness functions used during generation for the three goals identified above. We have evaluated our framework, EvoSuiteFIT, on a set of Java case examples. EvoSuiteFIT techniques attain significant improvements for two of the three goals, and show limited improvements on the third when the number of generations of evolution is fixed. Additionally, for two of the three goals, EvoSuiteFIT detects faults missed by the other techniques. The ability to adjust fitness functions allows strategic choices that efficiently produce more effective test suites, and examining these choices offers insight into how to attain our testing goals. We find that adaptive fitness function selection is a powerful technique to apply when an effective fitness function does not already exist for achieving a testing goal.

show abstract

Choosing the fitness function for the job: Automated generation of test suites that detect real faults

Cited by 31 publications

References 60 publications

Learning How to Search: Generating Exception-Triggering Tests Through Adaptive Fitness Function Selection

Learning How to Search: Generating Exception-Triggering Tests Through Adaptive Fitness Function Selection

Testing Method for Software With Randomness Using Genetic Algorithm

Learning how to search: generating effective test cases through adaptive fitness function selection

Contact Info

Product

Resources

About