Coverage Criteria for Behavioural Testing of Software Product Lines

Abstract

Abstract. Featured Transition Systems (FTS) is a mathematical structure to represent the behaviour of software product line in a concise way. The combination of the well-known transition systems approach to formal behavioural modelling with feature expressions was pivotal to the design of efficient verification approaches. Such approaches indeed avoid to consider products' behaviour independently, leading to often exponential savings. Building on this successful structure, we lay the foundations of model-based testing approach to SPLs. We define several FTS-aware coverage criteria and report on our experience combining FTS with usage-based testing for configurable websites.

Statistical prioritization for software product line testing: an experience report

“…This can help testers finding opportunities for test reuse since we demonstrated that a given behaviour can in fact be executed by many configurations. We also noticed the influence of variability models in the discrimination power of prioritization and reported on ways to cope with such a situations by either using additional coverage criteria [23,24] or ultimately relying on tester expertise. Indeed, as for sampling configuration from feature models, statistical testing of SPLs is constrained by multiple objectives [31].…”

“…Practically, we will build a second F T S which will represent only the behaviour of the SPL appearing in the valid finite traces generated from the usage model. This FTS' represents a prioritized subset of the original FTS that may be used to generate test cases [23]. Figure 4 presents the algorithm used to build an f ts from a set of traces (filtered during the algorithm) and a f ts.…”

“…Finally, the tool generates abstract test cases (i.e. sequences of actions) [23] using statistical testing algorithms on the usage model [25,62]. Those abstract test cases will have to be refined by providing inputs and expected outputs for the different actions [57].…”

On Featured Transition Systems

“…Since FTS are extensions of transition systems, a natural research direction was to consider "usual" coverage criteria (e.g., all-states, all-transitions) for product-line test generation [21]. In our work, we modelled test cases in terms of sequences of actions.…”

“…and the number of possible ones in the FTS, if the value is 1 then we obtain all-X coverage, where X is the set of elements under consideration for this coverage criterion. When such elements involve transitions, we impose that these transitions are executable by at least one product (see [21] for formal definitions). In our coffee machine, the following test suite both satisfies all-states and all-transitions coverage:…”