Many-Objective Test Suite Generation for Software Product Lines

Hierons, Robert M.; Li, Miqing; Liu, Xiaohui; Parejo, José Antonio; Segura, Sergio; Yao, Xin

doi:10.1145/3361146

Cited by 25 publications

(26 citation statements)

References 72 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For problems with more objectives, the parallel coordinates plot (instead of the scatter plot) is a helpful tool, which can reflect the convergence and diversity of a solution set to some extent [73]. It has started to be used recently, e.g., in [53], [89], [91], [143], [144].…”

Section: Aided Evaluation Methodsmentioning

confidence: 99%

See 1 more Smart Citation

How to Evaluate Solutions in Pareto-Based Search-Based Software Engineering: A Critical Review and Methodological Guidance

Chen

Yao

2022

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

Section: Aided Evaluation Methodsmentioning

confidence: 99%

“…This is obvious, as a violation of dependency implies faulty and incorrect configuration, thus valueless in practice. A similar situation applies to the test case generation problem [53], [103], [104], [124], [150] where the DM is typically interested in test suites with full coverage.…”

Section: When the Dm's Preferences Are Clearmentioning

confidence: 99%

How to Evaluate Solutions in Pareto-Based Search-Based Software Engineering: A Critical Review and Methodological Guidance

Chen

Yao

2022

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

“…While our model does not tie to any specific multiobjective optimizer, we use NSGA-II for both MMO and PMO in this work, because (1) it has been predominately used for software configuration tuning in prior work when multiple performance attributes are of interest [18,20,37,55,56]; (2) it shares many similarities with the SOGA that we compare in this work. Note that MMO may not be able to work with some multi-objective optimizers specifically designed for SBSE problems where the objectives are not treated equally, such as [28,29,50].…”

Section: Tuning Settingsmentioning

confidence: 99%

Multi-objectivizing software configuration tuning

Chen

2021

Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

Self Cite

View full text Add to dashboard Cite

Automatically tuning software configuration for optimizing a single performance attribute (e.g., minimizing latency) is not trivial, due to the nature of the configuration systems (e.g., complex landscape and expensive measurement). To deal with the problem, existing work has been focusing on developing various effective optimizers. However, a prominent issue that all these optimizers need to take care of is how to avoid the search being trapped in local optima -a hard nut to crack for software configuration tuning due to its rugged and sparse landscape, and neighboring configurations tending to behave very differently. Overcoming such in an expensive measurement setting is even more challenging. In this paper, we take a different perspective to tackle this issue. Instead of focusing on improving the optimizer, we work on the level of optimization model. We do this by proposing a meta multi-objectivization model (MMO) that considers an auxiliary performance objective (e.g., throughput in addition to latency). What makes this model unique is that we do not optimize the auxiliary performance objective, but rather use it to make similarly-performing while different configurations less comparable (i.e. Pareto nondominated to each other), thus preventing the search from being trapped in local optima.Experiments on eight real-world software systems/environments with diverse performance attributes reveal that our MMO model is statistically more effective than state-of-the-art single-objective counterparts in overcoming local optima (up to 42% gain), while using as low as 24% of their measurements to achieve the same (or better) performance result. CCS CONCEPTS• Software and its engineering → Software performance; Software configuration management and version control systems.

show abstract

“…This is obvious, as violation of dependency implies faulty and incorrect configuration, thus valueless in practice. A similar situation applies to the test case generation problem [46], [96], [97], [114], [138] where the DM is typically interested in test suites with full coverage. In addition, the DM may only be interested in solutions which reach certain level on some objective.…”

Section: When the Dm's Preferences Are Clearmentioning

confidence: 99%

How to Evaluate Solutions in Pareto-based Search-Based Software Engineering? A Critical Review and Methodological Guidance

Li,

Chen,

Yao

2020

Preprint

Self Cite

View full text Add to dashboard Cite

With modern requirements, there is an increasing tendancy of considering multiple objectives/criteria simultaneously in many Software Engineering (SE) scenarios. Such a multi-objective optimization scenario comes with an important issue -how to evaluate the outcome of optimization algorithms, which typically is a set of incomparable solutions (i.e., being Pareto non-dominated to each other). This issue can be challenging for the SE community, particularly for practitioners of Search-Based SE (SBSE). On one hand, multiobjective optimization may still be relatively new to SE/SBSE researchers, who may not be able to identify right evaluation methods for their problems. On the other hand, simply following the evaluation methods for general multiobjective optimisation problems may not be appropriate for specific SE problems, especially when the problem nature or decision maker's preferences are explicitly/implicitly available. This has been well echoed in the literature by various inappropriate/inadequate selection and inaccurate/misleading uses of evaluation methods. In this paper, we carry out a critical review of quality evaluation for multiobjective optimization in SBSE. We survey 717 papers published between 2009 and 2019 from 36 venues in 7 repositories, and select 97 prominent studies, through which we identify five important but overlooked issues in the area. We then conduct an in-depth analysis of quality evaluation indicators and general situations in SBSE, which, together with the identified issues, enables us to provide a methodological guidance to selecting and using evaluation methods in different SBSE scenarios.

show abstract

Many-Objective Test Suite Generation for Software Product Lines

Cited by 25 publications

References 72 publications

How to Evaluate Solutions in Pareto-Based Search-Based Software Engineering: A Critical Review and Methodological Guidance

How to Evaluate Solutions in Pareto-Based Search-Based Software Engineering: A Critical Review and Methodological Guidance

Multi-objectivizing software configuration tuning

How to Evaluate Solutions in Pareto-based Search-Based Software Engineering? A Critical Review and Methodological Guidance

Contact Info

Product

Resources

About