Location pairs: a test coverage metric for shared-memory concurrent programs

Abstract

We present a coverage metric targeted at shared-memory concurrent programs: the Location Pairs (LP) coverage metric. The goals of this metric are (i) to measure how thoroughly a program has been tested from a concurrency standpoint, i.e., whether enough qualitatively different thread interleavings have been explored, and (ii) to guide testing towards unexplored concurrency scenarios. This metric was inspired by an access pattern known to lead to high-level concurrency errors in industrial software and in the literature. We built a monitoring tool to measure LP coverage of test programs. We used the LP metric for interactive debugging, and compared LP coverage with other concurrency coverage metrics on Java benchmarks. We demonstrated that LP coverage corresponds better to concurrency errors, is a better measure of how well a program is exercised concurrency-wise by a test set, reaches saturation later than other coverage metrics, and is viable and useful as an interactive testing and debugging tool.

The Impact of Concurrent Coverage Metrics on Testing Effectiveness

“…The study most similar to the one we present in this paper is by Tasiran et al [19], who evaluate the locationpair metric empirically, and compare it to two other coverage metrics (method-pair and def-use) with respect to the correlation between coverage and fault finding ability. The study uses two case examples and generates faulty versions via concurrent mutation operators and manual fault seeding.…”

“…These include metrics defined over memory addresses or exhaustive sets of interleavings (e.g., all-du-path [7], ALL, SVAR [5]) and the series of extended coverage metrics proposed by Sherman et al [22]. Accesspair [22] and location-pair [19] are omitted as they are almost equivalent to the PSet metric. We interpret the LR-Def metric as generating two test requirements for read accesses: one for the use of memory defined by a local thread and the other for the use of memory defined by any remote thread.…”

“…For a give metric M , each test suite S's coverage is computed as the ratio of M 's test requirements that are satisfied by S to the total [6], sync-pair [15] Def-Use [19] number of test requirements satisfied across all executions for a given program version. We construct test executions while holding random test generation parameters constant (see Section III-B); because different parameters can result in covering different requirements, the coverage of M 's requirements is often less than 100%, and our measurements reflect this.…”

Development of Data Integrity Testing Tool for Mission-Critical Systems

“…A study conducted by [4] shows a test coverage metric targeting simultaneous programs that use shared memories. The objectives of this metric are measuring how programs that access a shared memory simultaneously should be tested from the viewpoint of concurrency control and guiding to make a concurrency control scenario that would not be found through testing.…”

Are concurrency coverage metrics effective for testing: a comprehensive empirical investigation

“…The study most similar to the one we present in this paper is by Tasiran et al , who evaluate the location‐pair metric empirically, and compare it to two other coverage metrics ( method‐pair and def‐use ) with respect to the correlation between coverage and fault detection. The study uses two case examples and generates faulty versions via concurrency mutation operators and manual fault seeding.…”