Comparing mutation coverage against branch coverage in an industrial setting

Abstract

The state-of-the-practice in software development is driven by constant change fueled by continuous integration servers. Such constant change demands for frequent and fully automated tests capable to detect faults immediately upon project build. As the fault detection capability of the test suite becomes so important, modern software development teams continuously monitor the quality of the test suite as well. However, it appears that the state-of-the-practice is reluctant to adopt strong coverage metrics (namely mutation coverage), instead relying on weaker kinds of coverage (namely branch coverage). In this paper, we investigate three reasons that prohibit the adoption of mutation coverage in a continuous integration setting: (1) the difficulty of its integration into the build system, (2) the perception that branch coverage is "good enough", and (3) the performance overhead during the build. Our investigation is based on a case study involving four open source systems and one industrial system. We demonstrate that mutation coverage reveals additional weaknesses in the test suite compared to branch coverage, and that it is able to do so with an acceptable performance overhead during project build.

Quantitative Metrics for Performance Monitoring of Software Code Analysis Accredited Testing Laboratories

“…Code coverage is a metric used to identify the quantity or set of “elements” of a software code that was tested by a given test case suite. These elements are, for example, lines of code, instructions, blocks of code, functions and procedures and can consider both source code and executable code [ 36 , 65 ]. Throughout this article, we consider the code coverage based on lines of software source code, which is a metric traditionally used to assess the test cases effectiveness [ 14 ].…”

“…Test cases contain a set of input values, pre-execution conditions, expected results and post-execution conditions developed for a specific objective and condition [ 36 , 65 ]. In the case of evaluating the quality of a software test case (or test suite), the classic use of code coverage is based on the premise that, the greater is the coverage of a test suite, the better is its quality—since this means that most of the software has been run (and therefore tested) [ 66 ].…”