A Comparative Study of Software Model Checkers as Unit Testing Tools: An Industrial Case Study

Abstract

Abstract-Conventional testing methods often fail to detect hidden flaws in complex embedded software such as device drivers or file systems. This deficiency incurs significant development and support/maintenance cost for the manufacturers. Model checking techniques have been proposed to compensate for the weaknesses of conventional testing methods through exhaustive analyses. Whereas conventional model checkers require manual effort to create an abstract target model, modern software model checkers remove this overhead by directly analyzing a target C program, and can be utilized as unit testing tools. However, since software model checkers are not fully mature yet, they have limitations according to the underlying technologies and tool implementations, potentially critical issues when applied in industrial projects. This paper reports our experience in applying Blast and CBMC to testing the components of a storage platform software for flash memory. Through this project, we analyzed the strong and weak points of two different software model checking technologies in the viewpoint of real-world industrial application-counterexample-guided abstraction refinement with predicate abstraction and SAT-based bounded analysis.Index Terms-Embedded software verification, software model checking, bounded model checking, CEGAR-based model checking, flash file systems.

Incremental test case generation using bounded model checking: an application to automatic rating

“…Furthermore, the possibility of application of model checkers for automated program testing is receiving increasing attention in the literature; see, for instance, [1,3,9,[11][12][13][14]. The strongest point of model checking is its capacity for the fully automated generation of test suites (see, for instance, [3]); on the other hand, some drawbacks resulting from the use of model checkers in test case generation are discussed in [9], originating in the fact that software model checking is not in a fully mature state yet.…”

“…A slightly different approach, where the model checker is used directly for unit testing, is presented in [13]. Finally, the authors of [14] provide a comparative study of applying different model checkers to component testing, from the viewpoint of a real-world industrial project.…”

Experience of improving the blast static verification tool

“…2. Such a representation of assume is not new (see, for example, [8]); however, its support is conditioned namely by the flexible repre sentation of the program control flow.…”

Automated Analysis of Industrial Embedded Software

“…Blast failed to detect the error and raised false alarms due to its limitations on handling bitwise operators and nested data structures. CBMC detected this error in 12 minutes with consuming 3 Gbyte of memory on average (details of the experiments can be found in [11]). In these analyses, however, we found that both Blast and CBMC had limitations for complex embedded C programs.…”

“…Furthermore, concrete applications of such techniques can guide new research directions to solve practical limitations observed in the studies. In this paper, we share our experience of applying various tools of model checking, software model checking, and concolic testing to flash memory storage platform [9,11,10] and smartphone platform [12].…”