How to Evaluate Solutions in Pareto-Based Search-Based Software Engineering: A Critical Review and Methodological Guidance

Li, Miqing; Chen, Tao; Yao, Xin

doi:10.1109/tse.2020.3036108

Cited by 52 publications

(39 citation statements)

References 146 publications

(328 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this paper, we replicate and extend their work [89] by further assessing CoGEE's effectiveness using the most recent best practice in search-based software engineering [6], [45], [67], [83] and predictive models for software engineering [88], [97], [109].…”

Section: Introductionmentioning

confidence: 85%

“…On the other hand, the Pareto Front's quality indicators allow us to quantify the overall quality of prediction models by measuring the trade-off between multiple competing objective values (in our case, SAE and CI). These indicators are well-known in the multi-objective optimisation literature [14], [19], [113] and have been extensively used in pre-vious software engineering work to evaluate and compare multi-and single-objective algorithms performance as an alternative to using the average values (see e.g., [35], [41], [42], [43], [67], [85]).…”

Section: Validation and Evaluation Criteriamentioning

confidence: 99%

“…As done in the original work, we use three Pareto Front quality indicators, namely Contributions (I C ), Hypervolume (I HV ), and Generational Distance (I GD ) [34], [67], as these three indicators complement each other in quantifying the overall quality of the prediction models in the objective space in terms of convergence, spread, uniformity, and cardinality [67].…”

Section: Validation and Evaluation Criteriamentioning

confidence: 99%

“…A large I HV indicates that the solutions produced by the approach covered more of the objective space. Hypervolume is one of the few quality indicators that can evaluate a solution set's quality in terms of all the four quality aspects including convergence, spread, uniformity, and cardinality [67]. Zitzler [118] demonstrated that Hypervolume measure is also strictly 'Pareto compliant' which means that Hypervolume of A is higher than B, if the Pareto set of A dominates that of B.…”

Section: Validation and Evaluation Criteriamentioning

confidence: 99%

“…Zitzler [118] demonstrated that Hypervolume measure is also strictly 'Pareto compliant' which means that Hypervolume of A is higher than B, if the Pareto set of A dominates that of B. To compute the I HV , we used [1.01, 1.01] as the reference point, which is the boundary of the optimisation problem after normalization with an added 1% offset, so that the boundary solutions could contribute to the I HV value [67].…”

Section: Validation and Evaluation Criteriamentioning

confidence: 99%

See 4 more Smart Citations

Multi-Objective Software Effort Estimation: A Replication Study

Tawosi

Sarro

Petrozziello

et al. 2022

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Replication studies increase our confidence in previous results when the findings are similar each time, and help mature our knowledge by addressing both internal and external validity aspects. However, these studies are still rare in certain software engineering fields. In this paper, we replicate and extend a previous study, which denotes the current state-of-the-art for multi-objective software effort estimation, namely CoGEE. We investigate the original research questions with an independent implementation and the inclusion of a more robust baseline (LP4EE), carried out by the first author, who was not involved in the original study. Through this replication, we strengthen both the internal and external validity of the original study. We also answer two new research questions investigating the effectiveness of CoGEE by using four additional evolutionary algorithms (i.e., IBEA, MOCell, NSGA-III, SPEA2) and a well-known Java framework for evolutionary computation, namely JMetal (rather than the previously used R software), which allows us to strengthen the external validity of the original study. The results of our replication confirm that: (1) CoGEE outperforms both baseline and state-of-the-art benchmarks statistically significantly (p < 0.001); (2) CoGEE's multi-objective nature makes it able to reach such a good performance; (3) CoGEE's estimation errors lie within claimed industrial human-expert-based thresholds. Moreover, our new results show that the effectiveness of CoGEE is generally not limited to nor dependent on the choice of the multi-objective algorithm. Using CoGEE with either NSGA-II, NSGA-III, or MOCell produces human competitive results in less than a minute. The Java version of CoGEE has decreased the running time by over 99.8% with respect to its R counterpart. We have made publicly available the Java code of CoGEE to ease its adoption, as well as, the data used in this study in order to allow for future replication and extension of our work.

show abstract

Section: Introductionmentioning

confidence: 85%

Section: Validation and Evaluation Criteriamentioning

confidence: 99%

Section: Validation and Evaluation Criteriamentioning

confidence: 99%

Section: Validation and Evaluation Criteriamentioning

confidence: 99%

Section: Validation and Evaluation Criteriamentioning

confidence: 99%

See 3 more Smart Citations

Multi-Objective Software Effort Estimation: A Replication Study

Tawosi

Sarro

Petrozziello

et al. 2022

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

show abstract

Quantum-inspired multi-objective African vultures optimization algorithm with hierarchical structure for software requirement

Liu,

Zhou,

Zhou

et al. 2024

Cluster Comput

View full text Add to dashboard Cite

Optimal priority assignment for real-time systems: a coevolution-based approach

et al. 2022

View full text Add to dashboard Cite

In real-time systems, priorities assigned to real-time tasks determine the order of task executions, by relying on an underlying task scheduling policy. Assigning optimal priority values to tasks is critical to allow the tasks to complete their executions while maximizing safety margins from their specified deadlines. This enables real-time systems to tolerate unexpected overheads in task executions and still meet their deadlines. In practice, priority assignments result from an interactive process between the development and testing teams. In this article, we propose an automated method that aims to identify the best possible priority assignments in real-time systems, accounting for multiple objectives regarding safety margins and engineering constraints. Our approach is based on a multi-objective, competitive coevolutionary algorithm mimicking the interactive priority assignment process between the development and testing teams. We evaluate our approach by applying it to six industrial systems from different domains and several synthetic systems. The results indicate that our approach significantly outperforms both our baselines, i.e., random search and sequential search, and solutions defined by practitioners. Our approach scales to complex industrial systems as an offline analysis method that attempts to find near-optimal solutions within acceptable time, i.e., less than 16 hours.

show abstract

How to Evaluate Solutions in Pareto-Based Search-Based Software Engineering: A Critical Review and Methodological Guidance

Cited by 52 publications

References 146 publications

Multi-Objective Software Effort Estimation: A Replication Study

Multi-Objective Software Effort Estimation: A Replication Study

Quantum-inspired multi-objective African vultures optimization algorithm with hierarchical structure for software requirement

Optimal priority assignment for real-time systems: a coevolution-based approach

Contact Info

Product

Resources

About