Online Testing of Nondeterministic Systems with the Reactive Planning Tester

Abstract

We describe a model-based construction of an online tester for black-box testing. Contemporary model-based online test generators focusing mainly on computationally cheap but far from optimal planning strategies cover just a fraction of the wide spectrum of test control strategies. Typical examples of those used are simple random choice and anti-ant. Exhaustive planning during online testing of nondeterministic systems looks out of reach because of the low scalability of the methods in regard to the model size. The reactive planning tester (RPT) studied in this chapter is targeted to fill the gap between these two extremes. The key idea of RPT lies in offline static analysis of the IUT (implementation under test) model to prepare the data and constraints for efficient online reactive planning. The external behavior of the IUT is modelled as an output observable nondeterministic EFSM (extended finite state machine) with the assumption that all the transition paths are feasible. A test purpose is attributed to the transitions of the IUT model by a set of Boolean variables called traps that are used to measure the progress of the test run. We present a way to construct a tester that at runtime selects a suboptimal test path from trap to trap by finding the shortest path that covers unvisited traps within planning horizon. The principles of reactive planning are implemented in the form of the decision rules of selecting the shortest paths at runtime. Based on an industrial scale case study, namely the city lighting system controller, we demonstrate the practical use of the RPT for systems with high degree of nondeterminism, deep nested control loops, and requiring strictly bounded tester response time. Tuning the planning horizon of the RPT allows a trade-off to be found between close to optimal test length and scalability of tester behavior with computationally feasible expenses.

Constraint-Based Heuristic On-line Test Generation from Non-deterministic I/O EFSMs

“…In this section we compare different strategies for generating test sequences for specific test goals. In particular, we compare the performance of χRPT with other search strategies such as the RPT off-line algorithm [4,15] and the randomized version of the anti-ant strategy [9]. As χRPT can be viewed as heuristic explicit-state forward reachability analysis rather than the symbolic analyzis done in the RPT, we have also chosen an explicit-state model checking tool UPPAAL [7] with modelling language close to EFSM for comparison.…”

“…Although test generation and execution from EFSMs has been studied extensively, on-line test generation from non-deterministic models tends to confine itself to computationally inexpensive but inefficient strategies such as random search or antiant. In this paper, we proposed a constraint-based heuristic approach (Heuristic Reactive Planning Tester (χRPT)) for I/O EFSM-based on-line test generation that could be used when the RPT on-line algorithm [15,4] is not applicable. χRPT is based on reachability constraints and properties of the underlying control graph of the I/O EFSM.…”

“…A widespread approach to modeling SUTs for test generation is using either Finite State Machines (FSMs) or Extended Finite State Machines (EFSMs) [8,1,15]. Test generation that includes the input data generation from EFSM models has been handled with different methods, including evolutionary algrithms [5], scenarios [17] and symbolic techniques [15]. The formal symbolic framework and notation of conformance for models involving data components is handled in [2,16].…”

“…The formal symbolic framework and notation of conformance for models involving data components is handled in [2,16]. On-line methods for test generation from non-deterministic models have also been studied by various authors [12,11,9,15].…”

“…Although there is a variety of approaches for test generation from EFSMs, most of the methods are not applicable or tend to be inefficient when applied to non-deterministic and industrial-scale systems for on-line test generation for specified test goals (e.g., coverage criteria). In this paper we propose an Heuristic Reactive Planning Tester (χRPT) to improve the scalability and performance of Reactive Planning Tester [15,4] by an heuristic constraint-driven on-line test generation technique. The only approaches we are aware of that have comparable goals are presented in [1,6].…”