Efficiency and early fault detection with lower and higher strength combinatorial interaction testing

Kästner

Proceedings of the 36th International Conference on Software Engineering

2014

In plugin-based systems, plugin conflicts may occur when two or more plugins interfere with one another, changing their expected behaviors. It is highly challenging to detect plugin conflicts due to the exponential explosion of the combinations of plugins (i.e., configurations). In this paper, we address the challenge of executing a test case over many configurations. Leveraging the fact that many executions of a test are similar, our variability-aware execution runs common code once. Only when encountering values that are different depending on specific configurations will the execution split to run for each of them. To evaluate the scalability of variability-aware execution on a large real-world setting, we built a prototype PHP interpreter called Varex and ran it on the popular WordPress blogging Web application. The results show that while plugin interactions exist, there is a significant amount of sharing that allows variabilityaware execution to scale to 2 50 configurations within seven minutes of running time. During our study, with Varex, we were able to detect two plugin conflicts: one was recently reported on WordPress forum and another one was not previously discovered.

Section: Scalability -Rq2mentioning

confidence: 99%

Exploring variability-aware execution for testing plugin-based web applications

Kästner

Proceedings of the 36th International Conference on Software Engineering

2014

IEICE Trans. Inf. & Syst.

“…Recently, Petke et al [21] presented a variant of the APCC λ metric, namely the Average Percentage of Coveringarray Coverage metric, which is similar to the APCCV λ metric. It does not require that all possible value combinations should be covered by the interaction test sequence S .…”

Section: Conclusion and Discussionmentioning

confidence: 99%

New Metrics for Prioritized Interaction Test Suites

Huang

Towey

Chen

et al. 2014

SUMMARYCombinatorial interaction testing has been well studied in recent years, and has been widely applied in practice. It generally aims at generating an effective test suite (an interaction test suite) in order to identify faults that are caused by parameter interactions. Due to some constraints in practical applications (e.g. limited testing resources), for example in combinatorial interaction regression testing, prioritized interaction test suites (called interaction test sequences) are often employed. Consequently, many strategies have been proposed to guide the interaction test suite prioritization. It is, therefore, important to be able to evaluate the different interaction test sequences that have been created by different strategies. A well-known metric is the Average Percentage of Combinatorial Coverage (shortly APCC λ ), which assesses the rate of interaction coverage of a strength λ (level of interaction among parameters) covered by a given interaction test sequence S . However, APCC λ has two drawbacks: firstly, it has two requirements (that all test cases in S be executed, and that all possible λ-wise parameter value combinations be covered by S ); and secondly, it can only use a single strength λ (rather than multiple strengths) to evaluate the interaction test sequence -which means that it is not a comprehensive evaluation. To overcome the first drawback, we propose an enhanced metric Normalized APCC λ (NAPCC) to replace the APCC λ . Additionally, to overcome the second drawback, we propose three new metrics: the Aver-

2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST)

“…Here is a long (yet partial) list of just some of the testing problems with citations to a few example papers (of many) that adopt an SBST approach to find suitable test data: functional testing [118], safety testing [11], [32], security testing [41], robustness testing [104], integration testing [18], [26], service-based testing [24], temporal testing [19], [113], [119], exception testing [114], Combinatorial Interaction Testing (CIT) [20], [25], [95], (and Software Product Line (SPL) testing [48]), state [77] and state-based-model testing [30], [78] (including popular modelling notations such as MATLAB Simulink [90], [129]), and mutation based test [37], [49] and mutant [65], [92] generation. The State of the Art: SBST has made many achievements, and demonstrated its wide applicability and increasing uptake.…”

Section: A Brief History Of Sbstmentioning

confidence: 99%

Achievements, Open Problems and Challenges for Search Based Software Testing

Harman¹,

Jia²,

Zhang³

2015

Self Cite

163

Abstract-Search Based Software Testing (SBST) formulates testing as an optimisation problem, which can be attacked using computational search techniques from the field of Search Based Software Engineering (SBSE). We present an analysis of the SBST research agenda 1 , focusing on the open problems and challenges of testing non-functional properties, in particular a topic we call 'Search Based Energy Testing' (SBET), Multi-objective SBST and SBST for Test Strategy Identification. We conclude with a vision of FIFIVERIFY tools, which would automatically find faults, fix them and verify the fixes. We explain why we think such FIFIVERIFY tools constitute an exciting challenge for the SBSE community that already could be within its reach.