Code coverage measurement is an important element in white-box testing, both in industrial practice and academic research. Other related areas are highly dependent on code coverage as well, including test case generation, test prioritization, fault localization, and others. Inaccuracies of a code coverage tool sometimes do not matter that much but in certain situations they can lead to serious confusion. For Java, the prevalent approach to code coverage measurement is to use bytecode instrumentation due to its various benefits over source code instrumentation. However, if the results are to be mapped back to source code this may lead to inaccuracies due to the differences between the two program representations. In this paper, we systematically investigate the amount of differences in the results of these two Java code coverage approaches, enumerate the possible reasons and discuss the implications on various applications. For this purpose, we relied on two widely used tools to represent the two approaches and a set of benchmark programs from the open source domain.

Many software testing elds, like white-box testing, test case generation, test prioritization and fault localization, depend on code coverage measurement. If used as an overall completeness measure, the minor inaccuracies of coverage data reported by a tool do not matter that much; however, in certain situations they can lead to serious confusion. For example, a code element that is falsely reported as covered can introduce false condence in the test. This work investigates code coverage measurement issues for the Java programming language. For Java, the prevalent approach to code coverage measurement is using bytecode instrumentation due to its various benets over source code instrumentation. As we have experienced, bytecode instrumentation-based code coverage tools produce dierent results than source code instrumentation-based ones in terms of the reported items as covered. We report on an empirical study to compare the code coverage results provided by tools using the dierent instrumentation types for Java coverage measurement on the method level. In particular, we want to nd out how much a bytecode instrumentation approach is inaccurate compared to a source code instrumentation method. The dierences are systematically investigated both in quantitative (how much the outputs dier) and in qualitative terms (what the causes for the dierences are). In addition, the impact on test prioritization and test suite reduction a possible application of coverage measurement is investigated in more detail as well. Keywords Code coverage • white-box testing • Java bytecode instrumentation • source code instrumentation • coverage tools • empirical study The nal publication is available at Springer via

No abstract

No abstract

Code coverage is successfully used to guide white box test design and evaluate the respective test completeness. However, simple overall coverage ratios are often not precise enough to effectively help when a (regression) test suite needs to be reassessed and evolved after software change. We present an approach for test suite assessment and improvement that utilizes code coverage information, but on a more detailed level and adds further evaluation aspects derived from the coverage. The main use of the method is to aid various test suite evolution situations such as removal, refactoring and extension of test cases as a result of code change or test suite efficiency enhancement. We define various metrics to express different properties of test suites beyond simple code coverage ratios, and present the assessment and improvement process as an iterative application of different improvement goals and more specific sub-activities. The method is demonstrated by applying it to improve the tests of one of our experimental systems.

Assessing the overall quality (adequacy for a particular purpose) of existing test suites is a complex task. Their code coverage is a simple yet powerful attribute for this purpose, so the additional benefits of mutation analysis may not always justify the comparably much higher costs and complexity of the computation. Mutation testing methods and tools slowly start to reach a maturity level at which their use in everyday industrial practice becomes possible, yet it is still not completely clear in which situations they provide additional insights into various quality attributes of the test suites. This paper reports on an experiment conducted on four open source systems' test suites to compare them from the viewpoints of code coverage, mutation score and test suite reducibility (the amount test adequacy is degraded in a reduced test suite). The purpose of the comparison is to find out when the different attributes provide additional insights with respect to defect density, a separately computed attribute for the estimation of real faults. We demonstrate that in some situations code coverage might be a sufficient indicator of the expected defect density, but mutation and reducibility are better in most of the cases.

Regression test suites of evolving software systems play a key role in maintaining software quality throughout continuous changes. They need to be effective (in terms of detecting faults and helping their localization) and efficient (optimally sized and without redundancy) at the same time. However, test suite quality attributes are usually difficult to formalize and measure. In this paper, we rely on a recent approach for test suite assessment and improvement that utilizes code coverage information, but at a more detailed level, hence it adds further evaluation aspects derived from the coverage. The basic idea of the method is to decompose the test suite and the program code into coherent logical groups which are easier to analyze and understand. Several metrics are then computed from code coverage information to characterize the test suite and its constituents. We extend our previous study and employ derived coverage metrics (which express efficiency and uniqueness) to analyze the test suite of a large scale industrial open source system containing 27 000 test cases.

No abstract