Integrating formal verification and conformance testing for reactive systems

Abstract

Abstract-In this paper, we describe a methodology integrating verification and conformance testing. A specification of a systeman extended input-output automaton, which may be infinite-state-and a set of safety properties ("nothing bad ever happens") and possibility properties ("something good may happen") are assumed. The properties are first tentatively verified on the specification using automatic techniques based on approximated state-space exploration, which are sound, but, as a price to pay for automation, are not complete for the given class of properties. Because of this incompleteness and of state-space explosion, the verification may not succeed in proving or disproving the properties. However, even if verification did not succeed, the testing phase can proceed and provide useful information about the implementation. Test cases are automatically and symbolically generated from the specification and the properties and are executed on a black-box implementation of the system. The test execution may detect violations of conformance between implementation and specification; in addition, it may detect violation/satisfaction of the properties by the implementation and by the specification. In this sense, testing completes verification. The approach is illustrated on simple examples and on a Bounded Retransmission Protocol.

Runtime Monitoring of Message-Based Workflows with Data

“…This is different from the quantification over message content used by LTL-FO + . A similar remark applies to the Input-Output State Transition Systems (IOSTS) used in [8] to specify monitoring properties. Although an IOSTS models input and output, the data parameters are not named and therefore cannot be quantified.…”

Combining Techniques to Verify Service-based Components

“…Many works that combine tests and proofs use finite state machines dialect as modelling DSL (Constant et al, 2007;Falzon and Pace, 2012;Artho et al, 2005). In the spirit of Model Based Testing (MBT), the authors focus on conformance checking and the goal is to generate test cases from a formal specification to check whether an implementation conforms to the model (Constant et al, 2007) or to monitor runtime verification (Falzon and Pace, 2012).…”

“…In the spirit of Model Based Testing (MBT), the authors focus on conformance checking and the goal is to generate test cases from a formal specification to check whether an implementation conforms to the model (Constant et al, 2007) or to monitor runtime verification (Falzon and Pace, 2012). Herber et al generate conformance tests to complete the model-checking of SystemC designs (Herber et al, 2009).…”

Runtime Enforcement of Web Service Message Contracts with Data

“…Timed automata [34] can be used to represent such timing dependencies between events; they otherwise behave like classical automata and cannot quantify over data elements other than time; they hence cannot be used in the present context. Extended timed automata, used in model checkers like Uppaal [35], have data variables in addition to clocks; however, it is unclear how, for example, the variable (and potentially unbounded) list of item IDs in a shopping cart could be encoded with data variables in order to express Runtime Properties 3 and 4; a similar remark applies to the Input-Output State Transition Systems (IOSTS) used in [36] to specify monitoring properties. In addition, the overwhelming majority of properties of commercial web services elicit constraints on data elements such as shopping carts or item IDs, and seldom refer to time.…”