No abstract

Despite much recent interest in randomised testing (fuzzing) of compilers, the practical impact of fuzzer-found compiler bugs on real-world applications has barely been assessed. We present the first quantitative and qualitative study of the tangible impact of miscompilation bugs in a mature compiler. We follow a rigorous methodology where the bug impact over the compiled application is evaluated based on (1) whether the bug appears to trigger during compilation; (2) the extent to which generated assembly code changes syntactically due to triggering of the bug; and (3) whether such changes cause regression test suite failures, or whether we can manually find application inputs that trigger execution divergence due to such changes. The study is conducted with respect to the compilation of more than 10 million lines of C/C++ code from 309 Debian packages, using 12% of the historical and now fixed miscompilation bugs found by four state-of-the-art fuzzers in the Clang/LLVM compiler, as well as 18 bugs found by human users compiling real code or as a by-product of formal verification efforts. The results show that almost half of the fuzzer-found bugs propagate to the generated binaries for at least one package, in which case only a very small part of the binary is typically affected, yet causing two failures when running the test suites of all the impacted packages. User-reported and formal verification bugs do not exhibit a higher impact, with a lower rate of triggered bugs and one test failure. The manual analysis of a selection of the syntactic changes caused by some of our bugs (fuzzer-found and non fuzzer-found) in package assembly code, shows that either these changes have no semantic impact or that they would require very specific runtime circumstances to trigger execution divergence. CCS Concepts: • Software and its engineering → Compilers; Software verification and validation.

A large amount of research has been carried out to automate white-box testing. While a wide range of different and sometimes heterogeneous code-coverage criteria have been proposed, there exists no generic formalism to describe them all, and available test automation tools usually support only a small subset of them. We introduce a new specification language, called HTOL (Hyperlabel Test Objectives Language), providing a powerful generic mechanism to define a wide range of test objectives. HTOL comes with a formal semantics, and can encode all standard criteria but full mutations. Besides specification, HTOL is appealing in the context of test automation as it allows handling criteria in a unified way

Databases are ubiquitous in software and testing of programs manipulating databases is thus essential to enhance the reliability of software. In this paper, we describe a clean and unified approach to automatically generate test inputs for such database programs. First, we propose a formal language, called ImperDB, to model database programs. ImperDB allows to model common program behaviors and data structures, as well as typical interaction scenarios between programs and databases. Secondly, we present a static analysis technique to generate test inputs for ImperDB programs, according to any chosen structural adequacy criterion. The technique considers an ImperDB program as a sequence of operations over a set of relational variables, modeling both the database original content and the program inputs. The problem of finding test inputs forcing the execution of a given path can then be transformed into the problem of solving constraints over the relational variables associated to the program. These constraints are expressed with the Alloy language and solved by the Alloy analyzer.

In so-called constraint-based testing, symbolic execution is a common technique used as a part of the process to generate test data for imperative programs. Databases are ubiquitous in software and testing of programs manipulating databases is thus essential to enhance the reliability of software. This work proposes and evaluates experimentally a symbolic execution algorithm for constraint-based testing of database programs. First, we describe SimpleDB, a formal language which offers a minimal and well-defined syntax and semantics, to model common interaction scenarios between programs and databases. Secondly, we detail the proposed algorithm for symbolic execution of SimpleDB models. This algorithm considers a SimpleDB program as a sequence of operations over a set of relational variables, modeling both the database tables and the program variables. By integrating this relational model of the program with classical static symbolic execution, the algorithm can generate a set of path constraints for any finite path to test in the controlflow graph of the program. Solutions of these constraints are test inputs for the program, including an initial content for the database. When the program is executed with respect to these inputs, it is guaranteed to follow the path with respect to which the constraints were generated. Finally, the algorithm is evaluated experimentally using representative SimpleDB models.