Automatic test case generation: what if test code quality matters?

Abstract

Test case generation tools that optimize code coverage have been extensively investigated. Recently, researchers have suggested to add other non-coverage criteria, such as memory consumption or readability, to increase the practical usefulness of generated tests. In this paper, we observe that test code quality metrics, and test cohesion and coupling in particular, are valuable candidates as additional criteria. Indeed, tests with low cohesion and/or high coupling have been shown to have a negative impact on future maintenance activities. In an exploratory investigation we show that most generated tests are indeed affected by poor test code quality. For this reason, we incorporate cohesion and coupling metrics into the main loop of search-based algorithm for test case generation. Through an empirical study we show that our approach is not only able to generate tests that are more cohesive and less coupled, but can (i) increase branch coverage up to 10% when enough time is given to the search and (ii) result in statistically shorter tests.

Testing with Fewer Resources: An Adaptive Approach to Performance-Aware Test Case Generation

“…Palomba et al [15] incorporated test cohesion and coupling metrics as secondary objectives within the preference criterion of MOSA to produce more maintainable test cases, from a developer point of view. Their approach produces more cohesive and less coupled test cases without reducing coverage.…”

“…While most of the research effort has been devoted to maximizing various code coverage criteria [2,6,8,9], recent work showed that further factors need to be considered for the generation of test cases [10][11][12][13]. Specifically, recent research investigated additional factors such as data input readability [11], test readability [13,14], test code quality [15], test diversity [16], execution time [12], and memory usage [10]. An early attempt to reduce the resource demand of generated tests is the work by Lakhotia et al [10].…”

“…While the aforementioned works showed the feasibility of lowering the cost (e.g., runtime) of the generated tests, they all pose two significant challenges on the full code coverage [19]. First, empirical results showed that combining coverage with non-coverage criteria is harmful to the final coverage compared to traditional strategies that target coverage only [10,15,18,19]. These approaches implement the classic one-branch-at-a-time (or single-target) approach, which consists in running bi-objective meta-heuristics (e.g., GA) multiple times, once for every branch in the code, while performance aspects are other dimensions to optimize for each branch separately.…”

“…Despite the theoretical benefits, the precise measurement of memory and runtime costs adds considerable overhead since it requires running each test case multiple times [24]. Consequently, there is a need for approaches that neither penalize the final coverage nor the fault detection capability of generated tests, while minimizing the test resource demand [15,19].…”

“…Recent work in the field explored alternative ways to integrate orthogonal objectives into the fitness function, which are based on the idea of using non-coverage aspects as a second-tier objective [15]. To address our second challenge, pDynaMOSA extends DynaMOSA [27] -the most recent and effective many-objective genetic algorithm for test case generation-by using the performance proxies as second-tier objectives while code branches are the first-tier objectives.…”