TSTL: the template scripting testing language

Holmes, Josie; Groce, Alex; Pinto, Jervis; Mittal, Pranjal; Azimi, Pooria; Kellar, Kevin; O’Brien, James F.

doi:10.1007/s10009-016-0445-y

Cited by 21 publications

(11 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More practically, we would like to tightly integrate our approach with automated test generation tools, such as TSTL , DeepState , Echidna and Manticore . Automated test generation tools tend to produce a large number of passing and failing tests and are often used in ‘overnight’ runs where adding a step to use mutants to triage failing tests and provide localization suggestions for the highly ranked tests may not even impose a noticeable overhead on current usage practices.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Using mutants to help developers distinguish and debug (compiler) faults

Holmes

Groce

2020

Software Testing Verif & Rel

Self Cite

View full text Add to dashboard Cite

Summary Measuring the distance between two program executions is a fundamental problem in dynamic analysis of software and useful in many test generation and debugging algorithms. This paper proposes a metric for measuring distance between executions and specializes it to an important application: determining similarity of failing test cases for the purpose of automated fault identification and localization in debugging based on automatically generated compiler tests. The metric is based on a causal concept of distance where executions are similar to the degree that changes in the program itself, introduced by mutation, cause similar changes in the correctness of the executions. Specifically, if two failing test cases for a compiler can be made to pass by applying the same mutant, those two tests are more likely to be due to the same fault. We evaluate our metric using more than 50 faults and 2,800 test cases for two widely used real‐world compilers and demonstrate improvements over state‐of‐the‐art methods for fault identification and localization. A simple operator selection approach to reducing the number of mutants can reduce the cost of our approach by 70%, while producing a gain in fault identification accuracy. We additionally show that our approach, although devised for compilers, is applicable as a conservative fault localization algorithm for other types of programs and can help triage certain types of crashes found in fuzzing non‐compiler programs more effectively than a state‐of‐the‐art technique.

show abstract

Section: Discussionmentioning

confidence: 99%

“…The work of Groce et al. provides an alternative approach to fault identification, based on term rewriting to normalize failures, but is not currently applicable to compiler fuzzing, as it only supports tests that are sequences of method calls in Python .…”

Section: Related Workmentioning

confidence: 99%

Using mutants to help developers distinguish and debug (compiler) faults

Holmes

Groce

2020

Software Testing Verif & Rel

Self Cite

View full text Add to dashboard Cite

show abstract

“…One goal of DeepState is to make it possible to easily use the same tool to perform both "unit-test" like testing (as above) and API/library random testing [15], [29], [20], where a test is a sequence of method/function calls chosen by the tool, not the developer. To assess the feasibility of using DeepState in this way, we created API tests for TestFS [36], a userlevel implementation of an ext3-like filesystem.…”

Section: File System Examplementioning

confidence: 99%

“…DeepState also targets the same space as property-based testing tools such as QuickCheck [10], ScalaCheck [28], Hypothesis [26], and TSTL [17], [20], but DeepState's test harnesses look like C/C++ unit tests. The major difference from previous tools is that DeepState aims to provide a front-end that can make use of a growing variety of backend methods for test generation, including (already) multiple binary analysis engines and a non-symbolic fuzzer.…”

Section: Introductionmentioning

confidence: 99%

“…call. The set of pathnames, however, is stored in an array, and there is a "memory" of previously used paths, a style of test construction known as pool-based [6], [29], [20]. While using pools can result in generating data that is not used, it makes it easier for symbolic execution engines and (especially) fuzzers to re-use a pathname (which is extremely important in most file system testing [15]).…”

mentioning

confidence: 99%

See 1 more Smart Citation

DeepState: Symbolic Unit Testing for C and C++

Goodman¹,

Groce²

2018

Proceedings 2018 Workshop on Binary Analysis Research

Self Cite

View full text Add to dashboard Cite

Unit testing is a popular software development methodology that can help developers detect functional regressions, explore boundary conditions, and document expected behavior. However, writing comprehensive unit tests is challenging and time-consuming, and developers seldom explore the obscure (and bug-hiding) corners of software behavior without assistance. DeepState is a tool that provides a Google Test-like API to give C and C++ developers pushbutton access to symbolic execution engines, such as Manticore and angr, and fuzzers, such as Dr. Fuzz. Rather than learning multiple complex tools, users learn one interface for defining a test harness, and can use various methods to automatically generate tests for software. In addition to providing a familiar interface to binary analysis and fuzzing for parameterized unit testing, DeepState also provides constructs that aid in the construction of API-sequence tests, where the tool chooses the functions or methods to call, allowing for even more diverse and powerful tests. By serving as a front-end to multiple tools, DeepState additionally provides a way to apply (novel) high-level strategies to test generation, and to compare effectiveness and efficiency of testing back-ends, including binary analysis tools.

show abstract

Emerging Aspects of Software Fault Localization

Tse¹,

Gorce

et al. 2023

Handbook of Software Fault Localization

View full text Add to dashboard Cite

TSTL: the template scripting testing language

Cited by 21 publications

References 66 publications

Using mutants to help developers distinguish and debug (compiler) faults

Using mutants to help developers distinguish and debug (compiler) faults

DeepState: Symbolic Unit Testing for C and C++

Emerging Aspects of Software Fault Localization

Contact Info

Product

Resources

About