An Improved Conformance Testing Method

Abstract

Abstract. In this paper, we present a novel conformance test suite derivation method. Similar to the HIS method, our method uses harmonized state identifiers for state identification and transition checking and can be applied to any reduced possibly partial deterministic or nondeterministic specification FSM. However, in contrast with the HIS method, in the proposed method appropriate state identifiers are selected on-the-fly (for transition checking) in order to shorten the length of the obtained test suite. Application examples and experimental results are provided. These results show that the proposed method generates shorter test suites than the HIS method. Particularly, on average, the ratio of the length of the test suites derived using the proposed method over the length of corresponding suites derived using the HIS method is 0.66 (0.55) when the number of states of an implementation equals to (is greater than) the number of states of the specification. These ratios are almost independent of the size of specifications.Many FSM-based test derivation methods have been developed for conformance testing of communication protocols and other reactive systems [2,3,10,12,14,15,17] [1,6,13,16]. In [2,3,10,14,15,17] testing methods, one usually assumes that not only the specification, but also the implementation can be modeled as a deterministic FSM, while in [7,8] the specification and the implementation are modeled as non-deterministic FSMs (NFSMs). If the behavior of a (deterministic/nondeterministic) implementation FSM is different than the specified behavior, the implementation contains a fault.The above methods, each provides the following fault coverage guarantee: If the specification can be modeled by a (reduced) FSM with n states and if a corresponding implementation can be modeled by an FSM with at most m states, where m is larger or equal to n, then a test suite can be derived by the method (for this given m) and the implementation passes this test suite if and only if it conforms (i.e. is equivalent) to the specification. A test suite is called m-complete [11] if it detects any nonconforming implementation with at most m states. Guessing the bound of m is an intuitive process based on the knowledge of a specification, the class of implementations which have to be tested for conformance and their interior structure [1]. All of the above methods assume that a reliable reset is available for each

A survey on formal active and passive testing with applications to the cloud

“…For FSM based conformance testing, there are a lot of testing methods such as W [18], Wp [25], DS [28], UIO [67], UIOv [80], HSI [62] and H [20] In general, networked systems are often updated. Thus, their specifications might often be changed.…”

Fault domain-based testing in imperfect situations: a heuristic approach and case studies

“…In this context, several test generation methods, such as W (Vasilevskii 1973;Chow 1978), Wp (Fujiwara et al 1991), HSI (Luo et al 1995), H (Dorofeeva et al 2005), and SPY (Simao et al 2009), have been proposed and used in system/software testing, such as testing of protocols and Web-based systems. These methods are fault domain-based since they produce test sets aiming at covering all faults in a given fault model or domain.…”

“…The aforementioned assumptions are required by most of methods found in the literature, such as W (Vasilevskii 1973;Chow 1978), Wp (Fujiwara et al 1991), HSI (Luo et al 1995), H (Dorofeeva et al 2005), and SPY (Simao et al 2009). The W method (Vasilevskii 1973;Chow 1978) is considered the seminal work on these methods.…”

“…To do so, recent methods rely on identifying separating sequences that will have less impact on the test size while the test set is built (this strategy is called on-the-fly). The proposal of on-the-fly methods are backed up by sufficient conditions that determine whether an arbitrary test set has the required properties or not (Dorofeeva et al 2005;Hierons and Ural 2006;Simao and Petrenko 2010). It was observed that recent methods rely on fewer and longer sequences to reduce the test set, while traditional methods show many short sequences (Endo and Simao 2013).…”

Uma estratégia para redução de conjuntos de sequências de teste para máquinas de estados finitos

“…Dentre os métodos mais conhecidos, pode-se destacar os métodos W (Chow, 1978), DS (Gonenc, 1970), UIO (Anido e Cavalli, 1995;Sabnani e Dahbura, 1988), Wp (Fujiwara et al, 1991), HSI (Luo et al, 1994;Petrenko et al, 1993), H (Dorofeeva et al, 2005a) e State Counting (Petrenko e Yevtushenko, 2005).…”