Online Testing of LTL Properties for Java Code

Abstract

Abstract. LTL specifications are commonly used in runtime verification to describe the requirements about the system behavior. Efficient techniques derive, from LTL specifications, monitors that can check if system executions respect these properties. In this paper we present an online testing approach which is based on LTL properties of Java programs. We present an algorithm able to derive and execute test cases from monitors for LTL specifications. Our technique actively tests a Java class, avoids false failures, and it is able to check the correctness of the outputs also in the presence of nondeterminism. We devise several coverage criteria and strategies for visiting the monitors, providing different qualities in terms of test size, testing time, and fault detection capability.

Automated Requirements-Based Testing of Black-Box Reactive Systems

“…In the third setting we inject faults manually. The results we obtained with our experiments show that our approach can outperform the one in [3] by finding more induced faults. Furthermore, generating tests based on the specification can be as effective as approaches based on the system model, discovering almost the same number of faults.…”

“…The research most closely related to ours is presented in [3] where the authors describe a methodology for online testing of Java classes. Their key technique is to exploit a monitor derived from LTL specifications to check conformance of the system to stated requirements, with a focus on safety properties.…”

“…Runtime verification [9] techniques can be seen as a form of oracle-based testing [10]: each test is executed on the system implementation and the test oracle, i.e., the monitor in runtime verification jargon, observes the system and checks whether its executions are behaviors allowed by the specification or not. Following this stream of research, a technique based on the use of monitors as test oracles is proposed in [3]. Their approach can test for safety properties ("something bad will never happen"), but it does not deal with liveness properties ("something good will happen infinitely often").…”

“…Our approach is inspired by [3], but aims to deal with a more general class of properties. Our methodology is based on a visit of the Büchi automaton corresponding to the requirements.…”

“…We evaluate our approach in three different experimental settings. In the first one we consider benchmarks taken from the LTL Track of the 2018 edition of the Reactive Synthesis Competition (SYNTCOMP 2018) 1 and we compare our approach with the one described in [3]. In the second setting we use the Adaptive Cruise Control (ACC) prototype implemented in [2] and we compare the tests generated by our approach with those generated with a model-based generation strategy.…”