Performance Evaluation Metrics for Multi-Objective Evolutionary Algorithms in Search-Based Software Engineering: Systematic Literature Review

Abstract

Many recent studies have shown that various multi-objective evolutionary algorithms have been widely applied in the field of search-based software engineering (SBSE) for optimal solutions. Most of them either focused on solving newly re-formulated problems or on proposing new approaches, while a number of studies performed reviews and comparative studies on the performance of proposed algorithms. To evaluate such performance, it is necessary to consider a number of performance metrics that play important roles during the evaluation and comparison of investigated algorithms based on their best-simulated results. While there are hundreds of performance metrics in the literature that can quantify in performing such tasks, there is a lack of systematic review conducted to provide evidence of using these performance metrics, particularly in the software engineering problem domain. In this paper, we aimed to review and quantify the type of performance metrics, number of objectives, and applied areas in software engineering that reported in primary studies—this will eventually lead to inspiring the SBSE community to further explore such approaches in depth. To perform this task, a formal systematic review protocol was applied for planning, searching, and extracting the desired elements from the studies. After considering all the relevant inclusion and exclusion criteria for the searching process, 105 relevant articles were identified from the targeted online databases as scientific evidence to answer the eight research questions. The preliminary results show that remarkable studies were reported without considering performance metrics for the purpose of algorithm evaluation. Based on the 27 performance metrics that were identified, hypervolume, inverted generational distance, generational distance, and hypercube-based diversity metrics appear to be widely adopted in most of the studies in software requirements engineering, software design, software project management, software testing, and software verification. Additionally, there are increasing interest in the community in re-formulating many objective problems with more than three objectives, yet, currently are dominated in re-formulating two to three objectives.

Research on hyper‐level of hyper‐heuristic framework for MOTCP

“…a: There are certain numbers of LLHs and their HD values in the current iteration. The HD probability of the ith LLH, denoted as pðiÞ, is calculated as Equation (7), in which HD i is the HD value of the ith LLH, P N i¼1 HD i is the sum of HD values of all LLHs.…”

“…Multi‐objective evolutionary algorithms (MOEAs) in search‐based software engineering [6] have been applied to solve MOTCP problems. Researches show that the performance of those algorithms may vary significantly for different testing scenarios [7]. However, in industrial applications, because of the cost, they usually select algorithms that are easier to implement and optimize.…”

Salience Based Stakeholders Selection Keeping Coverage in the Next Release Problem Solution

“…Our proposal is to group stakeholders based on salience, to reduce the number of stakeholders involved in a requirement selection process, while maintaining the agreed definition for NRP, so as to facilitate the application of abundant and validated algorithms to solve this problem [11,27], but with fewer data to manage. The metric we propose to quantify the value of salience is driven by stakeholder theory [10,23] (unlike other works [2,3,15] which, although they quantify stakeholders, based on many stakeholder attributes, are outside this framework) and is not an estimated value based on levels of influence (as in the cases of the works [31,30]).…”

“…We refer to the new optimisation problem as dNRP. After solving it (using any of the algorithms proposed in the literature [27]) we get a set of non-dominated solutions defining a Pareto front F Def .…”

“…After every change, the dominance is checked, substituting the solution if needed. Although it may not be the best algorithm to find the Pareto fronts (which is beyond the scope of this paper [27,41]), it is at least simple and obtains the Pareto fronts in the same way in all cases, avoiding biases that would occur in experiments if different methods were used to obtain solutions.…”