SpecTest: Specification-Based Compiler Testing

Abstract

Compilers are error-prone due to their high complexity. They are relevant for not only general purpose programming languages, but also for many domain specific languages. Bugs in compilers can potentially render all programs at risk. It is thus crucial that compilers are systematically tested, if not verified. Recently, a number of efforts have been made to formalise and standardise programming language semantics, which can be applied to verify the correctness of the respective compilers. In this work, we present a novel specification-based testing method named SpecTest to better utilise these semantics for testing. By applying an executable semantics as test oracle, SpecTest can discover deep semantic errors in compilers. Compared to existing approaches, SpecTest is built upon a novel test coverage criterion called semantic coverage which brings together mutation testing and fuzzing to specifically target less tested language features. We apply SpecTest to systematically test two compilers, i.e., the Java compiler and the Solidity compiler. SpecTest improves the semantic coverage of both compilers considerably and reveals multiple previously unknown bugs.

K-ST: A Formal Executable Semantics of PLC Structured Text Language

“…As discussed earlier, because ST compilers are typically provided by vendors, the execution behavior of compilers may be different, and may even be inconsistent with respect to the high-level semantics [49]. One of the main applications of the proposed semantics is to define the correct and secure 'reference' execution behavior of ST, which can help programmer detect bugs in existing ST compilers.…”

Generating Test Suites for GPU Instruction Sets through Mutation and Equivalence Checking

“…In a manner similar to our goal of using mutation testing to add more tests, SpecTest [26] monitors the executable model of a language extracted from its semantics to obtain coverage of small step semantics that are used by test programs. These test programs are then mutated to increase coverage of language semantics by injecting program constructs that correspond to unexercised semantics.…”