Performance robustness of AI planners in the 2014 International Planning Competition

Abstract

Solver competitions have been used in many areas of AI to assess the current state of the art and guide future research and development. AI planning is no exception, and the International Planning Competition (IPC) has been frequently run for nearly two decades. Due to the organisational and computational burden involved in running these competitions, solvers are generally compared using a single homogeneous hardware and software environment for all competitors. To what extent does the specific choice of hardware and software environment have an effect on solver performance, and is that effect distributed equally across the competing solvers? In this work, we use the competing planners and benchmark instance sets from the 2014 IPC to investigate these two questions. We recreate the 2014 IPC Optimal and Agile tracks on two distinct hardware environments and eight distinct software environments. We show that solver performance varies significantly based on the hardware and software environment, and that this variation is not equal for all planners. Furthermore, the observed variation is sufficient to change the competition rankings, including the top-ranked planners for some tracks.

Sparkle: Toward Accessible Meta-Algorithmics for Improving the State of the Art in Solving Challenging Problems

“…By utilising the Sparkle platform, we can organise Sparkle challenges, a novel type of competitive event, which aims to advance the state of the art in solving various challenging computational problems, including Boolean satisfiability (SAT) [46], AI planning [57] and other problems, by leveraging automatically constructed algorithm selectors and by quantifying contributions of individual solvers.…”

“…At the moment, traditional solver competitions (such as the international SAT competitions 6 [44], [45], [46], international planning competitions 14 [61], [57], [62], etc.) measure the performance of each individual solver across a large set of benchmark instances, and identify the winning solver(s) based on their overall performance across this instance set.…”

Reproducibility in Evolutionary Computation

“…Unreproducible Computational environment. Although lab conditions in computer science are very controlled, conclusions may depend on details of the computational environment such as the compiler, the hardware, or specific libraries [Bocchese et al 2018].…”

Reformulation Techniques for Automated Planning: A Systematic Review

“…The main purpose of this review is to provide a qualitative comparison of the existing reformulation approaches, with the aim of helping experts and practitioners in identifying the most promising technique to be used, and to highlight research gaps that can foster further research in the area. Note that empirical analysis of the reviewed reformulation techniques is beyond the scope of this work due to the high variability, among different papers, of considered benchmarks, planning engines, and software/hardware infrastructure [Bocchese et al, 2018].…”