Testing concurrent programs to achieve high synchronization coverage

Abstract

The effectiveness of software testing is often assessed by measuring coverage of some aspect of the software, such as its code. There is much research aimed at increasing code coverage of sequential software. However, there has been little research on increasing coverage for concurrent software. This paper presents a new technique that aims to achieve high coverage of concurrent programs by generating thread schedules to cover uncovered coverage requirements. Our technique first estimates synchronization-pair coverage requirements, and then generates thread schedules that are likely to cover uncovered coverage requirements. This paper also presents a description of a prototype tool that we implemented in Java, and the results of a set of studies we performed using the tool on a several open-source programs. The results show that, for our subject programs, our technique achieves higher coverage faster than random testing techniques; the estimation-based heuristic contributes substantially to the effectiveness of our technique.

Ask the Mutants: Mutating Faulty Programs for Fault Localization

“…We also plan to apply MUSE to larger subjects such as PHP with multiple test suites. In addition, we will apply the mutation idea to conconlic unit testing [30] and concurrent coverage-based testing [24].…”

The Impact of Concurrent Coverage Metrics on Testing Effectiveness

“…For a give metric M , each test suite S's coverage is computed as the ratio of M 's test requirements that are satisfied by S to the total [6], sync-pair [15] Def-Use [19] number of test requirements satisfied across all executions for a given program version. We construct test executions while holding random test generation parameters constant (see Section III-B); because different parameters can result in covering different requirements, the coverage of M 's requirements is often less than 100%, and our measurements reflect this.…”

“…Thus, to maximize the effectiveness of testing processes, researchers create test adequacy criteria based on these metrics, and develop techniques to quickly satisfy them. The development of such techniques has long been an active area of research in the context of structural coverage metrics for non-concurrent programs [8], [9], [10], [11], and as multithreaded programs have become more common the development of techniques centered around concurrent coverage metrics has also become an active area of research [12], [13], [14], [15].…”

Performance regression testing of concurrent classes

“…Another line of research to help detecting concurrency bugs are approaches to influence the scheduler by adding random delays [15], by exploring interleavings systematically [36,12] or randomly [7], by forcing schedules that expose potential concurrency bugs [45,40,32,30,26], and by forcing schedules that cover yet uncovered interleavings [25,57,49]. By influencing the scheduler, these approaches also influence the performance of a concurrent program, and therefore are not directly applicable in our approach.…”

“…To achieve correctness, developers can rely on bug finding techniques [17,16,6,46,26,41] and approaches that influence the schedule of concurrent executions [15,36,12,7,45,40,32,30,25,57,49]. In contrast, developers currently have only little support to measure, improve, and maintain the performance of thread-safe classes.…”