Software assurance by bounded exhaustive testing

Abstract

Abstract-Bounded exhaustive testing (BET) is a verification technique in which software is automatically tested for all valid inputs up to specified size bounds. A particularly interesting case of BET arises in the context of systems that take structurally complex inputs. Early research suggests that the BET approach can reveal faults in small systems with inputs of low structural complexity, but its potential utility for larger systems with more complex input structures remains unclear. We set out to test its utility on one such system. We used Alloy and TestEra to generate inputs to test the Galileo dynamic fault tree analysis tool, for which we already had both a formal specification of the input space and a test oracle. An initial attempt to generate inputs using a straightforward translation of our specification to Alloy did not work well. The generator failed to generate inputs to meaningful bounds. We developed an approach in which we factored the specification, used TestEra to generate abstract inputs based on one factor, and passed the results through a postprocessor that reincorporated information from the second factor. Using this technique, we were able to generate test inputs to meaningful bounds, and the inputs revealed nontrivial faults in the Galileo implementation, our specification, and our oracle. Our results suggest that BET, combined with specification abstraction and factoring techniques, could become a valuable addition to our verification toolkit and that further investigation is warranted.

Agile validation of model transformations using compound F-Alloy specifications

“…As an example, we have seen earlier that the Alloy language comes with a tool, the Alloy Analyzer, allowing the generation and visual rendering of instances as graphs (where atoms are nodes and links are edges). This tool and approach has been successfully applied in the validation of software systems [17,21], and generally contributed to close the gap between the engineer modeling the system in Alloy, and their client, the domain expert for which the system is designed [3]. However, it has been highlighted in [9] that despite recent advances in Alloy instances representations [25], intuitive visualization of large instances can only be achieved using the knowledge of their domain of application, namely domain specific visualization.…”

Reducing the Costs of Bounded-Exhaustive Testing

“…The experience with bounded-exhaustive testing in academia and industry shows that it can find faults in real code [4,3,5,11] but also produces a large number of failures. Identifying a few faults out of many failures is a challenging task.…”

“…Our focus is on boundedexhaustive testing [2,3,4,5,6,11] that tests the code for all inputs within given bounds. Previous work considered how to describe a set of inputs (using declarative [2,4] or imperative [5] approaches) and how to efficiently generate them.…”

“…One approach to reducing the cost is to automate testing. Bounded-exhaustive testing is an automated approach that checks the code under test for all inputs within given bounds [2,3,4,5,6]. The rationale is that many faults can be revealed within small bounds [7,8], and exhaustively testing within the bounds ensures that no "corner case" is missed.…”

“…Two key costs in this context are machine time for test generation and execution (which also translates into human time for waiting for these results [9,10]) and human time for inspection of failures. Previous experience shows that bounded-exhaustive testing can discover important faults [3,4,5,11] but also can have high costs. This paper proposes, and evaluates on a case study, three novel techniques that reduce these costs of bounded-exhaustive testing.…”