Learning how to search: generating effective test cases through adaptive fitness function selection

Abstract

Search-based test generation is guided by feedback from one or more fitness functions—scoring functions that judge solution optimality. Choosing informative fitness functions is crucial to meeting the goals of a tester. Unfortunately, many goals—such as forcing the class-under-test to throw exceptions, increasing test suite diversity, and attaining Strong Mutation Coverage—do not have effective fitness function formulations. We propose that meeting such goals requires treating fitness function identification as a secondary optimization step. An adaptive algorithm that can vary the selection of fitness functions could adjust its selection throughout the generation process to maximize goal attainment, based on the current population of test suites. To test this hypothesis, we have implemented two reinforcement learning algorithms in the EvoSuite unit test generation framework, and used these algorithms to dynamically set the fitness functions used during generation for the three goals identified above. We have evaluated our framework, EvoSuiteFIT, on a set of Java case examples. EvoSuiteFIT techniques attain significant improvements for two of the three goals, and show limited improvements on the third when the number of generations of evolution is fixed. Additionally, for two of the three goals, EvoSuiteFIT detects faults missed by the other techniques. The ability to adjust fitness functions allows strategic choices that efficiently produce more effective test suites, and examining these choices offers insight into how to attain our testing goals. We find that adaptive fitness function selection is a powerful technique to apply when an effective fitness function does not already exist for achieving a testing goal.

The integration of machine learning into automated test generation: A systematic mapping study

“…Many authors use ML to enhance existing test generation approaches-often based on Genetic Algorithms. Almulla and Gay use reinforcement learning to select which fitness functions will be optimized in service of a higherlevel testing goal [5,78,79]. For example, the agent can learn which combinations of fitness functions best trigger exceptions [5], increase input diversity [78], or increase Strong Mutation Coverage [79].…”

“…Almulla and Gay use reinforcement learning to select which fitness functions will be optimized in service of a higherlevel testing goal [5,78,79]. For example, the agent can learn which combinations of fitness functions best trigger exceptions [5], increase input diversity [78], or increase Strong Mutation Coverage [79]. He et al use reinforcement learning to improve coverage of private and inherited methods by augmenting generated tests [81].…”

Automatic Test Data Generation for Basis Path Testing

“…The following are some related works on test data generation fields: Hussein Almulla and Gregory Gay proposed a feedbackbased test data generation approach using multiple fitness functions. The method is guided by the choice of fitness function to produce more effective results (19) . Meena Sharma and Babita Pathak has proposed the crow search method with improved fitness function , that combines the branch distance and branch predicate to reach the critical paths.…”

Automated Support for Unit Test Generation: A Tutorial Book Chapter

“…a diverse test suite. For example, recent research uses reinforcement learning to adapt the strategy employed by a search-based algorithm to generate more diverse tests for particular classes-undertest [17]. A study comparing many different test techniques found that techniques focused on diversity were among the best possible in selecting a small set of test cases [18].…”

“…Reinforcement learning has also been used within search-based test generation frameworks to adapt the test generation strategy to particular systems or problems. For example, it has been applied to automatically tune parameters of the metaheuristic [29], to select fitness functions in multi-objective search in service of optimising a high-level goal (e.g., selecting fitness functions that cause a class to throw more exceptions) [17], and to transform test cases by substituting individual actions for alternatives that may assist in testing inheritance in class hierarchies or covering private code [30] Other researchers have proposed the use of supervised machine learning to generate test input (e.g., [31,32]). In such approaches, a set of existing test input and results of executing that input (either the output or some other result, such as the code coverage) are used to train a model.…”