TSTL: a language and tool for testing (demo)

Abstract

Writing a test harness is a difficult and repetitive programming task, and the lack of tool support for customized automated testing is an obstacle to the adoption of more sophisticated testing in industry. This paper presents TSTL, the Template Scripting Testing Language, which allows users to specify the general form of valid tests for a system in a simple but expressive language, and tools to support testing based on a TSTL definition. TSTL is a minimalist templatebased domain-specific language, using the source language of the Software Under Test (SUT) to support most operations, but adding declarative idioms for testing. TSTL compiles to a common testing interface that hides the details of the SUT and provides support for logging, code coverage, delta debugging, and other core testing functionality, making it easy to write universal testing tools such as random testers or model checkers that apply to all TSTL-defined harnesses. TSTL is currently available for Python, but easily adapted to other languages as well.

Using mutants to help developers distinguish and debug (compiler) faults

“…More practically, we would like to tightly integrate our approach with automated test generation tools, such as TSTL , DeepState , Echidna and Manticore . Automated test generation tools tend to produce a large number of passing and failing tests and are often used in ‘overnight’ runs where adding a step to use mutants to triage failing tests and provide localization suggestions for the highly ranked tests may not even impose a noticeable overhead on current usage practices.…”

“…The work of Groce et al. provides an alternative approach to fault identification, based on term rewriting to normalize failures, but is not currently applicable to compiler fuzzing, as it only supports tests that are sequences of method calls in Python .…”

Generic and Effective Specification of Structural Test Objectives

“…Test description languages. Some languages have been designed to support the implementation of test harnesses at the program (TSTL [39], UDITA [40]) or model (TTCN-3 [41], UML Testing Profile [42]) level. A test harness is the helper code that executes the testing process in practice, which notably includes test definition, documentation, execution and logging.…”

Provenance and Pseudo-Provenance for Seeded Learning-Based Automated Test Generation

“…We implemented a novel test generation technique for the TSTL [19,16,17,18] test generation language and tool for Python. In this approach, the seed tests are split into (usually short) subsequences of length k. In place of the usual algorithm for random testing, where a new test action is randomly chosen at each step during testing, our approach always attempts to follow some subsequence, in a best-effort fashion (if the next step in the current sub-sequence is not enabled, it is skipped).…”