Statistical Racing Techniques for Improved Empirical Evaluation of Evolutionary Algorithms

Yuan, Bo; Gallagher, Marcus

doi:10.1007/978-3-540-30217-9_18

Cited by 48 publications

(36 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some procedures for tackling the tuning problem have been proposed by Birattari [12]. Among them, the F-Race method is the best performing one and has been used in a number of works on metaheuristics [55,56,57,58,59].…”

Section: Literature Overviewmentioning

confidence: 99%

Out-of-the-Box and Custom Implementation of Metaheuristics. A Case Study: The Vehicle Routing Problem with Stochastic Demand

Pellegrini

Birattari

2011

Intelligent Computational Optimization in Engineering

View full text Add to dashboard Cite

Abstract. Metaheuristics are a class of effective algorithms for optimization problems. A basic implementation of a metaheuristic typically requires rather little development effort. With a significantly larger investment in the design, implementation, and fine-tuning, metaheuristics can often produce state-of-the-art results. According to the amount of development effort, we say that an implementation of a metaheuristic is either an out-of-the-box version or a custom one. The possibility of implementing metaheuristics in such a flexible way is one of the major strengths of these algorithms. Nonetheless, it also hides some possible catches. In particular, it should be noticed that results obtained with out-of-the-box implementations cannot be always generalized to custom ones, and vice versa. The goal of this analysis is to stress that these two ways of using metaheuristics are different. As a case study, we focus on the vehicle routing problem with stochastic demand and on five among the most successful metaheuristics-namely, tabu search, simulated annealing, genetic algorithms, iterated local search, and ant colony optimization. We show that the relative performance of these algorithms strongly varies whether one considers out-of-the-box implementations or custom ones, in which the parameters are accurately fine-tuned. Moreover, we underline the relevance of clearly stating the framework in which the results reported in the literature have been obtained. To this aim, we consider also an implementation of the same algorithms as described in the literature.

show abstract

Section: Literature Overviewmentioning

confidence: 99%

Out-of-the-Box and Custom Implementation of Metaheuristics. A Case Study: The Vehicle Routing Problem with Stochastic Demand

Pellegrini

Birattari

2011

Intelligent Computational Optimization in Engineering

View full text Add to dashboard Cite

show abstract

“…These approaches consider only the Pareto front set evaluation to improve MOEAs, whereas our approach evaluates operators contribution in improving fitness and thus injects mutation operators that are eligible to make MOEAs converge faster. In [37], the authors have shown that racing algorithms can be used to reduce the computational resources inherent from using evolutionary algorithms in large scale experimental studies, their approach automates solutions selection and discards solutions that do not introduce results improvement. Whereas racing techniques eliminate worst solutions candidates to speed up the search, in our approach we keep considering worst ranked candidates to maintain operators diversity.…”

Section: Related Workmentioning

confidence: 99%

Optimizing Multi-objective Evolutionary Algorithms to Enable Quality-Aware Software Provisioning

Kateb

Fouquet

Bourcier

et al. 2014

2014 14th International Conference on Quality Software

View full text Add to dashboard Cite

Abstract-Elasticity is a key feature for cloud infrastructures to continuously align allocated computational resources to evolving hosted software needs. This is often achieved by relaxing quality criteria, for instance security or privacy because quality criteria are often conflicting with performance. As an example, software replication could improve scalability and uptime while decreasing privacy by creating more potential leakage points. The conciliation of these conflicting objectives has to be achieved by exhibiting trade-offs. Multi-Objective Evolutionary Algorithms (MOEAs) have shown to be suitable candidates to find these trade-offs and have been even applied for cloud architecture optimizations. Still though, their runtime efficiency limits the widespread adoption of such algorithms in cloud engines, and thus the consideration of quality criteria in clouds. Indeed MOEAs produce many dead-born solutions because of the Darwinian inspired natural selection, which results in a resources wastage. To tackle MOEAs efficiency issues, we apply a process similar to modern biology. We choose specific artificial mutations by anticipating the optimization effect on the solutions instead of relying on the randomness of natural selection. This paper introduces the Sputnik algorithm, which leverages the past history of actions to enhance optimization processes such as cloud elasticity engines. We integrate Sputnik in a cloud elasticity engine, dealing with performance and quality criteria, and demonstrate significant performance improvement, meeting the runtime requirements of cloud optimization.

show abstract

“…Parameter tuning thus takes place before the run, e.g., exploiting the lessons learned from previous runs. Standard approaches from experimental studies such as ANOVA or Design Of Experiments have been used for parameter tuning, e.g., modeling the impact of parameter values on the overall performance and accordingly determining the optimal values [4,45,3,36]. These methods, however, are very computationally expensive, as each observation corresponds to the average of a few evolutionary runs; furthermore, static settings are usually considered (the parameter value is fixed along the run), whereas the optimal setting likely depends on the local landscape explored by the genetic population.…”

Section: Parameter Setting In Evolutionary Algorithmsmentioning

confidence: 99%

Analyzing bandit-based adaptive operator selection mechanisms

Fialho

Costa

Schoenauer

et al. 2010

Ann Math Artif Intell

133

123

View full text Add to dashboard Cite

Several techniques have been proposed to tackle the Adaptive Operator Selection (AOS) issue in Evolutionary Algorithms. Some recent proposals are based on the Multi-Armed Bandit (MAB) paradigm: each operator is viewed as one arm of a MAB problem, and the rewards are mainly based on the fitness improvement brought by the corresponding operator to the individual it is applied to. However, the AOS problem is dynamic, whereas standard MAB algorithms are known to optimally solve the exploitation versus exploration trade-off in static settings. An original dynamic variant of the standard MAB Upper Confidence Bound algorithm is proposed here, using a sliding time window to compute both its exploitation and exploration terms. In order to perform sound comparisons between AOS algorithms, artificial scenarios have been proposed in the literature. They are extended here toward smoother transitions between different reward settings. The resulting original testbed also includes a real evolutionary algorithm that is applied to the well-known Royal Road problem. It is used here to perform a thorough analysis of the behavior of AOS algorithms, to assess their sensitivity with respect to their own hyper-parameters, and to propose a sound comparison of their performances.

show abstract

Statistical Racing Techniques for Improved Empirical Evaluation of Evolutionary Algorithms

Cited by 48 publications

References 9 publications

Out-of-the-Box and Custom Implementation of Metaheuristics. A Case Study: The Vehicle Routing Problem with Stochastic Demand

Out-of-the-Box and Custom Implementation of Metaheuristics. A Case Study: The Vehicle Routing Problem with Stochastic Demand

Optimizing Multi-objective Evolutionary Algorithms to Enable Quality-Aware Software Provisioning

Analyzing bandit-based adaptive operator selection mechanisms

Contact Info

Product

Resources

About