The Oracle Problem When Testing from MSCs

Abstract

Message Sequence Charts (MSCs) form a popular language in which scenariobased specifications and models can be written. There has been significant interest in automating aspects of testing from MSCs. This paper concerns the Oracle Problem, in which we have an observation made in testing and wish to know whether this is consistent with the specification. We assume that there is an MSC specification and consider the case where we have entirely independent local testers (local observability) and where the observations of the local testers are logged and brought together (tester observability). It transpires that under local observability the Oracle Problem can be solved in low-order polynomial time if we use sequencing, loops and choices but becomes NP-complete if we also allow parallel components; if we place a bound on the number of parallel components then it again can be solved in polynomial time. For tester observability, the problem is NP-complete when we have either loops or choices. However, it can be solved in low-order polynomial time if we have only one loop, no choices, and no parallel components. If we allow parallel components then the Oracle Problem is NP-complete for tester observability even if we restrict to the case where there are at most two processes.

A small-step approach to multi-trace checking against interactions

“…In the following we demonstrate the NP-hardness of this membership problem through a reduction of the 1-in-3-SAT problem [20]. This discussion is inspired by [1,8,6].…”

“…Those theorems and all the definitions and lemmas they depend on have been encoded in the Gallina language so as to formally verify our proofs using the Coq automated theorem prover. A Coq proof, which includes the 2 previous demonstrations is available online 6 .…”

“…the difficulty in determining when to apply stimuli in order to realize a targeted global execution. Our work, however, falls within another domain which is Passive Testing [2,16] (in which testers are only observers), and discusses other problems such as the Test Oracle Problem [6] (determining expected outputs w.r.t. given stimuli).…”

“…Both local and global invariants can be checked although the latter is more costly in terms of computations. Our approach is different in that the reference for the analysis is not a logical property but a model of interaction as in [16,6]. [16] discusses passive testing against choreography models expressed in the language Chor [19].…”

“…the absence of weak sequencing and the nature of loops), (2) [16] only handles synchronous communication between services, which cannot always describe accurately concrete implementations and (3) the local logs are not directly checked against the model but first pass through a synthesis step in which a global log is reconstituted and then this global log is checked. Authors in [6] investigate the computational cost of log analysis w.r.t. graphs of MSCs.…”