Combining Model-Based Testing and Runtime Monitoring for Program Testing in the Presence of Nondeterminism

Abstract

Abstract-In case of underspecified or not fully predictable systems, models specifying system behaviors are nondeterministic. Nondeterminism poses several challenges for the validation and verification activities, including the problem of inconclusive tests in model-based testing with model checker. It is a validation technique that use model checker counterexamples as test cases.In this paper, we tackle the problem of testing nondeterministic systems by combining model-based testing and runtime conformance monitoring: the input sequences of the tests are automatically generated from nondeterministic models; then their execution is runtime monitored to check conformance of the code w.r.t. its specification. This technique provides an oracle for the test data, it never bears inconclusive responses, and it allows measuring the requirement coverage.The approach uses the Abstract State Machines as formal method for specification purposes and Java as implementation language. As a proof of concepts, the Tic-Tac-Toe game is taken as example of a system with nondeterministic behavior (both at specification and code levels).

Mysteries of DropBox: Property-Based Testing of a Distributed Synchronization Service

“…Arcaini et al [19] generate tests with a model checker in a similar way, but instead of introducing branches, they reuse the model as a run-time monitor, to check that even if the system follows a different path from the test case, then its input-output behavior still conforms to the model. They do assume that the inputs in the generated test can be supplied to the SUT even though it is following an unexpected path, and they assume that outputs from the SUT always provide enough information to uniquely identify the corresponding model state ('strong conformance').…”

“…Our monitoring state machine is deterministic, but includes unobservable transitions-eliding those transitions makes it non-deterministic. We allow multiple possible model states during monitoring ('weak conformance' in the sense of [19]), and we treat the SUT as a black box-there is no need for instrumentation of the implementation to connect it to our model. Our examples are more complex real applications, and we found a number of unexpected behaviors.…”

Integrating formal methods into medical software development: The ASM approach

“…This can be used by the designer to estimate if the validation activity is commensurate with the risk associated with the use of the software for the specified intended use. (9) Since V&V are performed by exploiting unambiguous mathematicalbased techniques, they facilitate independent evaluation of software quality assurance.…”

ASM2Bogor: An approach for verification of models specified through Asmeta language

“…Tic Tac Toe simulator is a game played on a 3×3 square board [29]. It lets to play the game with the computer.…”