Feature Interaction Faults Revisited: An Exploratory Study

Abstract

Abstract-While a large body of research is dedicated to testing for feature interactions in configurable software, there has been little work that examines what constitutes such a fault at the code level. In consequence, we do not know how prevalent real interaction faults are in practice, what a typical interaction fault looks like in code, how to seed interaction faults, or whether current interaction testing techniques are effective at finding the faults they aim to detect.We make a first step in this direction, by deriving a whitebox criterion for an interaction fault. Armed with this criterion, we perform an exploratory study on hundreds of faults from the field in two open source systems. We find that only three of the 28 which appear to be interaction faults are in fact due to features' interactions. We investigate the remaining 25 and find that, although they could have been detected without interaction testing, varying the system configuration amplifies the fault-finding power of a test suite, making these faults easier to expose. Thus, we characterize the benefits of interaction testing in regards to both interaction and non-interaction faults. We end with a discussion of several mutations that can be used to mimic interaction faults based on the faults we see in practice.

Beyond the rainbow: self-adaptive failure avoidance in configurable systems

“…Firefox is widely deployed, has a large bug database and automated testing framework and has been used in prior research on testing configurable systems [22]. It also has an automated testing framework, Mozmill [31], that comes with regression tests and can simulate clients running Firefox use cases.…”

“…To these we added seven that include assertions about which features are enabled, tests that give the same effect as a seeded configuration-dependent fault. Each assertion corresponds to a fault in version 18 whose report in the Firefox bug repository [33] included a reconfiguration workaround, some of which were taken from [22].…”

“…There is also research on testing highly-configurable system [10,11,22,27,37,39,45,45]. Many sampling techniques have been used for selecting a subset of configurations for testing such as covering arrays for sampling configurable systems [37,45] and software product lines [36], Reisner et al [38] use symbolic execution of configurable software systems to gather evidence that we have a much smaller feasible configuration space possible then previously thought.…”

“…Research on testing configurable systems has focused on a particular type of fault called a feature interaction fault. Such a fault triggers only under a specific combination of features [11,22,27,37,39,45]. The problem of feature interaction faults has existed for years [46].…”

A Technique for Agile and Automatic Interaction Testing for Product Lines

“…Garvin and Cohen 2011 [17] did an exploratory study on two open source product lines. They extracted 28 faults that could be analyzed and which was configuration dependent.…”