Code coverage for suite evaluation by developers

Abstract

One of the key challenges of developers testing code is determining a test suite's quality -its ability to find faults. The most common approach is to use code coverage as a measure for test suite quality, and diminishing returns in coverage or high absolute coverage as a stopping rule. In testing research, suite quality is often evaluated by a suite's ability to kill mutants (artificially seeded potential faults). Determining which criteria best predict mutation kills is critical to practical estimation of test suite quality. Previous work has only used small sets of programs, and usually compares multiple suites for a single program. Practitioners, however, seldom compare suites -they evaluate one suite. Using suites (both manual and automatically generated) from a large set of real-world open-source projects shows that evaluation results differ from those for suite-comparison: statement (not block, branch, or path) coverage predicts mutation kills best.

On the effectiveness of unit tests in test-driven development

“…A literature review by Shahid, Ibrahim, and Naz [38] proposes 12 different granularities of measuring code coverage, some of which are branch, instruction, line, cyclomatic complexity, instruction and method coverage. According to an empirical study [19], the use of coverage measures to determine the defect detection capability of a test suite could be dependent on the context. In industrial environments, statement coverage is found as the most effective predictor of test suite quality, although, branch coverage or some variants of path coverage may be more useful in research contexts [19].…”

“…According to an empirical study [19], the use of coverage measures to determine the defect detection capability of a test suite could be dependent on the context. In industrial environments, statement coverage is found as the most effective predictor of test suite quality, although, branch coverage or some variants of path coverage may be more useful in research contexts [19]. Another study by Tengeri et al [39] argues that the correlation between high code coverage and defect detection effectiveness of test suites is not always present or evident.…”

“…Gopinath et al [19] also argues that, in terms of defect detection ability, mutation testing could be considered as the most effective way to predict test suite effectiveness. Other coverage measures such as statement and branch coverage could also be correlated with mutation detection capability although these findings are dependent on the selection of a proper population of subjects and test suite.…”

“…Other coverage measures such as statement and branch coverage could also be correlated with mutation detection capability although these findings are dependent on the selection of a proper population of subjects and test suite. Following Gopinath et al's [19] claim that the development approach in which unit tests are generated could likely affect the interpretation of unit tests, we use mutation score indicator to measure the effectiveness of unit tests written in the context of two different development approaches.…”

An assessment of operational coverage as both an adequacy and a selection criterion for operational profile based testing

“…As discussed in the introduction, the effectiveness of coverage criteria is still a very active research topic (Gopinath et al 2014;Inozemtseva and Holmes 2014;Kochhar et al 2015;Staats et al 2012). However, the studies evaluating the effectiveness of coverage measures consider the faults (or mutations) as having same importance and do not take into account their respective probability of failure.…”