A platform for search-based testing of concurrent software

Křena, Bohuslav; Letko, Zdeněk; Vojnar, Tomáš; Ur, Shmuel

doi:10.1145/1866210.1866215

Cited by 27 publications

(24 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We used these controls because previous work indicates that they can impact the effectiveness of the testing process [13]. The specific values used were selected based on our previous experience in this domain [15] and pilot studies, both of which indicated that larger or finer grained delays and probabilities did not yield significantly different results.…”

Section: ) Test Generation and Executionmentioning

confidence: 99%

“…Thus, to maximize the effectiveness of testing processes, researchers create test adequacy criteria based on these metrics, and develop techniques to quickly satisfy them. The development of such techniques has long been an active area of research in the context of structural coverage metrics for non-concurrent programs [8], [9], [10], [11], and as multithreaded programs have become more common the development of techniques centered around concurrent coverage metrics has also become an active area of research [12], [13], [14], [15].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The Impact of Concurrent Coverage Metrics on Testing Effectiveness

Hong

Staats

Ahn

et al. 2013

2013 IEEE Sixth International Conference on Software Testing, Verification and Validation

View full text Add to dashboard Cite

Abstract-When testing multithreaded programs, the number of possible thread interactions makes exploring all interactions infeasible in practice. In response, researchers have developed concurrent coverage metrics for multithreaded programs. These metrics allow them to estimate how well they have exercised concurrent program behavior, just as branch and statement coverage metrics do for sequential program testing. However, unlike sequential coverage metrics, the effectiveness of concurrent coverage metrics in testing remains largely unexamined. In this paper, we explore the relationship between concurrent coverage and fault detection effectiveness by studying the application of eight concurrent coverage metrics in testing nine concurrent programs. Our results show that existing concurrent coverage metrics are often moderate to strong predictors of concurrent testing effectiveness, and are generally reasonable targets for test suite generation. Nevertheless, their relative effectiveness as predictors and test generation targets varies across programs, and thus additional work is needed in this area.

show abstract

Section: ) Test Generation and Executionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

The Impact of Concurrent Coverage Metrics on Testing Effectiveness

Hong

Staats

Ahn

et al. 2013

2013 IEEE Sixth International Conference on Software Testing, Verification and Validation

View full text Add to dashboard Cite

show abstract

“…This is motivated by the rise of multicore systems [Sutter and Larus 2005], the ineffectiveness of traditional testing for detecting and reproducing concurrency bugs due to nondeterminism [Křena et al 2010], and the desire for automatic, precise analysis, which is hard to achieve using static techniques [Bessey et al 2010]. …”

Section: Introductionmentioning

confidence: 99%

Concurrency Testing Using Controlled Schedulers

Thomson

Donaldson

Betts

2016

ACM Trans. Parallel Comput.

View full text Add to dashboard Cite

We present an independent empirical study on concurrency testing using controlled schedulers. We have gathered 49 buggy concurrent software benchmarks, drawn from public code bases, which we call SCTBench. We applied a modified version of an existing concurrency testing tool to SCTBench, testing five controlled scheduling techniques: depth-first search, preemption bounding, delay bounding, a controlled random scheduler, and probabilistic concurrency testing (PCT). We attempt to answer several research questions, including: Which technique performs the best, in terms of bug finding ability? How effective are the two main schedule bounding techniques, preemption bounding and delay bounding, at finding bugs? What challenges are associated with applying concurrency testing techniques to existing code? Can we classify certain benchmarks as trivial or non-trivial? Overall, we found that PCT (with parameter d=3) was the most effective technique in terms of bug finding; it found all the bugs found by the other techniques, plus an additional three, and it missed only one bug. Surprisingly, we found that the "naïve" controlled random scheduler, which randomly chooses one thread to execute at each scheduling point, performs well, finding more bugs than preemption bounding and just two fewer bugs than delay bounding. Our findings confirm that delay bounding is superior to preemption bounding and schedule bounding is superior to an unbounded depth-first search. The majority of bugs in SCTBench can be exposed using a small schedule bound (1-2), supporting previous claims, although one benchmark requires 5 preemptions. We found that the need to remove nondeterminism and control all synchronization (as is required for systematic concurrency testing) can be nontrivial. There were 8 distinct programs that could not easily be included in out study, such as those that perform network and inter-process communication. We report various properties about the benchmarks tested, such as the fact that the bugs in 18 benchmarks were exposed 50% of the time when using random scheduling. We note that future work should not use the benchmarks that we classify as trivial when presenting new techniques, other than as a minimum baseline. We have made SCTBench and our tools publicly available for reproducibility and use in future work.

show abstract

“…There are only few approaches that try to generate not only schedules, but also test cases: Krena et al [13] proposed a search-based technique that optimizes schedules used by the Contest [7] tool. Ballerina [19] uses random test prefixes, and then explores two methods executed in parallel in two different threads.…”

Section: B Concurrent Testingmentioning

confidence: 99%

Generating Unit Tests for Concurrent Classes

Steenbuck

2013

2013 IEEE Sixth International Conference on Software Testing, Verification and Validation

View full text Add to dashboard Cite

Abstract-As computers become more and more powerful, programs are increasingly split up into multiple threads to leverage the power of multi-core CPUs. However, writing correct multi-threaded code is a hard problem, as the programmer has to ensure that all access to shared data is coordinated. Existing automated testing tools for multi-threaded code mainly focus on re-executing existing test cases with different schedules. In this paper, we introduce a novel coverage criterion that enforces concurrent execution of combinations of shared memory access points with different schedules, and present an approach that automatically generates test cases for this coverage criterion. Our CONSUITE prototype demonstrates that this approach can reliably reproduce known concurrency errors, and evaluation on nine complex open source classes revealed three previously unknown data-races.

show abstract

A platform for search-based testing of concurrent software

Cited by 27 publications

References 30 publications

The Impact of Concurrent Coverage Metrics on Testing Effectiveness

The Impact of Concurrent Coverage Metrics on Testing Effectiveness

Concurrency Testing Using Controlled Schedulers

Generating Unit Tests for Concurrent Classes

Contact Info

Product

Resources

About