Learning How to Search: Generating Exception-Triggering Tests Through Adaptive Fitness Function Selection

Abstract

Search-based test generation is guided by feedback from one or more fitness functions-scoring functions that judge solution optimality. Choosing informative fitness functions is crucial to meeting the goals of a tester. Unfortunately, many goals-such as forcing the class-under-test to throw exceptions, increasing test suite diversity, and attaining Strong Mutation Coverage-do not have effective fitness function formulations. We propose that meeting such goals requires treating fitness function identification as a secondary optimization step. An adaptive algorithm that can vary the selection of fitness functions could adjust its selection throughout the generation process to maximize goal attainment, based on the current population of test suites. To test this hypothesis, we have implemented two reinforcement learning algorithms in the EvoSuite unit test generation framework, and used these algorithms to dynamically set the fitness functions used during generation for the three goals identified above. We have evaluated our framework, EvoSuiteFIT, on a set of real Java case examples. EvoSuiteFIT techniques attain significant improvements for two of the three goals, and show small improvements on the third when the number of generations of evolution is fixed. Additionally, for all goals, EvoSuiteFIT detects faults missed by the other techniques. The ability to adjust fitness functions allows Evo-SuiteFIT to make strategic choices that efficiently produce more effective test suites, and examining its choices offers insight into how to attain our testing goals. We find that AFFS is a powerful technique to apply when an effective fitness function does not already exist for generating tests to achieve a testing goal.

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

“…However, a model is generally static once trained and cannot be improved without re-training. Reinforcement learning (RL) algorithms select actions based on an estimation of their effectiveness towards achieving a measurable goal [5]. RL often does not require training data, instead learning through sequences of interactions with its environment.…”

Mapping the Structure and Evolution of Software Testing Research Over the Past Three Decades

“…There are connections to quality and reliability as they offer metrics that can be used as part of prediction or knowledge discovery. Automated test generation often makes use of ML [71,67]. ML techniques are also used as part of test case prioritization, security analysis, and fault injection analysis.…”

Understanding The Impact of Solver Choice in Model-Based Test Generation

“…In this study, we make use of 37 models from the open-source Lustre Benchmarks dataset 3 . This dataset has been used in previous test generation experiments [38], and includes complex models such as Docking_Approach, a NASA-created example that describes the behavior of a space shuttle as it docks with the International Space Station [24].…”

“…This dataset has been used in previous test generation experiments [38], and includes complex models such as Docking_Approach, a NASA-created example that describes the behavior of a space shuttle as it docks with the International Space Station [24]. Another model, Infusion_Manager represents 1 http://www.mathworks.com/products/simulink 2 http://www.mathworks.com/stateflow 3 Available from https://github.com/Greg4cr/Reworked-Benchmarks/tree/SingleNode. the prescription management of an infusion pump device [22][23][24].…”

“…Automated approaches to parameter tuning have been proposed in multiple domains, generally making use of some form of reinforcement learning [3,32]. A model-based test generation framework could perform a short parameter tuning experiment before generating a test suite.…”