How Effectively Is Defective Code Actually Tested?

Abstract

Background: Newspaper headlines still regularly report latent software defects. Such defects have often evaded testing for many years. It remains difficult to identify how well a system has been tested. It also remains difficult to assess how successful at finding defects particular tests are. Coverage and mutation testing are frequently used to asses test effectiveness. We look more deeply at the performance of commonly used JUnit testing by assessing how much JUnit testing was done and how effective that testing was at detecting defects in seven open source systems. Aim: We aim to identify whether defective code has been effectively tested by JUnit tests as non-defective code. We also aim to identify the characteristics of JUnit tests that are related to identifying defects. Methodology: We first extract the defects from seven open source projects using the SZZ algorithm. We match those defects with JUnit tests to identify the proportion of defects that were covered by JUnit tests. We also do the same for non-defective code. We then use Principal Component Analysis and machine learning to investigate the characteristics of JUnit tests that were successful in identifying defects. Results: Our findings suggest that most of the open source systems we investigated are under-tested. On average over 66% of defective methods were not linked to any JUnit tests. We show that the number of methods touched by a JUnit test is strongly related to that test uncovering a defect. Conclusion: More JUnit tests need to be produced for the seven open source systems that we investigate. JUnit tests need to be relatively sophisticated, in particular they should touch more than just one method during the test.

Large‐scale characterization of Java streams

“…In particular, the authors report that 61% of the analyzed tests contain multiple assertions, and nearly 78% of the assertTrue and assertFalse assertions lack an error message. Petrić et al 89 analyze the effectiveness of unit tests on 7 Java projects (out of a pool of 5508 candidates). They find that only about 30% of defective methods are covered by unit tests.…”

Which Software Faults Are Tests Not Detecting?

“…In line with this view, Bowes et al investigated the effectiveness of tests using metrics that capture various facets of testing [10]. Consequently, several studies have empirically demonstrated that some metrics, such as the number of asserts [42] and methods invoked [31] by a test, are highly associated with faults. In this work we analyse mutants to identify how tests can be improved to cover specific types of fault.…”

“…Other factors have also been investigated in terms of the ability of tests to uncover faults. Most notably, test suite size [19], number of assertions [42] and number of covered methods per test [31] have all been shown to have an effect on test effectiveness. The relationship between fault type and test effectiveness has not been adequately explored.…”

Identification of microservices from monolithic applications through topic modelling

“…Although it provides good results, it has the disadvantage of requiring inputs or test cases to properly execute the system. Unfortunately, this is not the case in most legacy systems [32].…”