Balancing Soundness and Efficiency for Practical Testing of Configurable Systems

Souto, Sabrina; d’Amorim, Marcelo; Gheyi, Rohit

doi:10.1109/icse.2017.64

Cited by 29 publications

(20 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Souto et al [76] explore the tension between soundness and efficiency for configurable systems. They do so by extending SPLat [44], a variability-aware dynamic technique that, for each test, monitors configuration variables accessed during test execution and change their values to run the test on new configurations, stopping either when no new configurations can be dynamically explored or a certain threshold is met.…”

Section: Selected Studiesmentioning

confidence: 99%

“…Drupal (PHP modules based Web content management system) Parejo et al [60] Multi-objective Drupal (PHP modules based Web content management system) Souto et al [76] random, one-enabled, one-disabled, most-enableddisabled and pairwise computed from SPLat [44] 8 small SPLs + GCC (50 most used options). Samplings' sizes and number of failures were considered.…”

Section: Pairwisementioning

confidence: 99%

See 1 more Smart Citation

Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack

et al. 2018

View full text Add to dashboard Cite

Many approaches for testing configurable software systems start from the same assumption: it is impossible to test all configurations. This motivated the definition of variability-aware abstractions and sampling techniques to cope with large configuration spaces. Yet, there is no theoretical barrier that prevents the exhaustive testing of all configurations by simply enumerating them if the effort required to do so remains acceptable. Not only this: we believe there is a lot to be learned by systematically and exhaustively testing a configurable system. In this case study, we report on the first ever endeavour to test all possible configurations of the industry-strength, open source configurable software system: JHipster, a popular code generator for web applications. We built a testing scaffold for the 26,000+ configurations of JHipster using a cluster of 80 machines during 4 nights for a total of 4,376 hours (182 days) CPU time. We find that 35.70% configurations fail and we identify the feature interactions that cause the errors. We show that sampling strategies (like A. Halin, G. Perrouin (FNRS research associate) PReCISE, dissimilarity and 2-wise): (1) are more effective to find faults than the 12 default configurations used in the JHipster continuous integration; (2) can be too costly and exceed the available testing budget. We cross this quantitative analysis with the qualitative assessment of JHipster's lead developers.

show abstract

Section: Selected Studiesmentioning

confidence: 99%

Section: Pairwisementioning

confidence: 99%

Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack

et al. 2018

View full text Add to dashboard Cite

show abstract

“…The authors of this paper recently found [16] that SPLat can be combined with sampling to balance cost of exploration with fault-detection ability. As future work, we plan to apply the ideas from that work to improve regression testing of configurable systems even further.…”

Section: Othermentioning

confidence: 99%

Time-space efficient regression testing for configurable systems

Souto

d’Amorim

2018

Journal of Systems and Software

Self Cite

View full text Add to dashboard Cite

Configurable systems are those that can be adapted from a set of options. They are prevalent and testing them is important and challenging. Existing approaches for testing configurable systems are either unsound (i.e., they can miss fault-revealing configurations) or do not scale. This paper proposes EvoSPLat, a regression testing technique for configurable systems. EvoSPLat builds on our previouslydeveloped technique, SPLat, which explores all dynamically reachable configurations from a test. EvoSPLat is tuned for two scenarios of use in regression testing: Regression Configuration Selection (RCS) and Regression Test Selection (RTS). EvoSPLat for RCS prunes configurations (not tests) that are not impacted by changes whereas EvoSPLat for RTS prunes tests (not configurations) which are not impacted by changes. Handling both scenarios in the context of evolution is important.Experimental results show that EvoSPLat is promising. We observed a substantial reduction in time (∼22%) and in the number of configurations (∼45%) for configurable Java programs. In a case study on a large real-world configurable system (GCC), EvoSPLat reduced ∼35% of the running time. Comparing EvoSPLat with sampling techniques, 2-wise was the most efficient technique, but it missed two bugs whereas EvoSPLat detected all bugs four times faster than 6-wise, on average.

show abstract

“…Besides those two cases (testing only default configurations or exhaustively testing all configurations), several approaches for testing configurable systems have been proposed [7,11,20,22,26,30,31,34]. Some of them consider only the feature model [7,22,26,31] in order to define products to be tested.…”

mentioning

confidence: 99%

“…However, they may explore configurations not reached by tests. Other approaches [20,30,34] take the code (test or source) into account in addition to the feature model, and dynamically explore all reachable configurations from a given test. However, such dynamic techniques only explore configurations related to testing.…”

mentioning

confidence: 99%

Testing configurable software systems

Ferreira

Viggiato

Souza

et al. 2020

Proceedings of the 24th ACM Conference on Systems and Software Product Line: Volume a - Volume A

View full text Add to dashboard Cite

Configurable software systems can be adapted or configured according to a set of features to increase reuse and productivity. The testing process is essential because configurations that fail may potentially hurt user experience and degrade the reputation of a project. However, testing configurable systems is very challenging due to the number of configurations to run with each test, leading to a combinatorial explosion in the number of configurations and tests. Currently, several testing techniques and tools have been proposed to deal with this challenge, but their potential practical application remains mostly unexplored. To encourage the research area on testing configurable systems, researchers and practitioners should be able to try out their solutions in common datasets. In this paper, we propose a dataset with 22 configurable software systems and an extensive test suite. Moreover, we report failures found in these systems and source code metrics to allow evaluating candidate solutions. We hope to engage the community and stimulate new and existing approaches to the problem of testing configurable systems.

show abstract

Balancing Soundness and Efficiency for Practical Testing of Configurable Systems

Cited by 29 publications

References 29 publications

Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack

Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack

Time-space efficient regression testing for configurable systems

Testing configurable software systems

Contact Info

Product

Resources

About