On the impact of the cutoff time on the performance of algorithm configurators

Abstract

Algorithm configurators are automated methods to optimise the parameters of an algorithm for a class of problems. We evaluate the performance of a simple random local search configurator (ParamRLS) for tuning the neighbourhood size k of the RLS k algorithm. We measure performance as the expected number of configuration evaluations required to identify the optimal value for the parameter. We analyse the impact of the cutoff time κ (the time spent evaluating a configuration for a problem instance) on the expected number of configuration evaluations required to find the optimal parameter value, where we compare configurations using either best found fitness values (ParamRLS-F) or optimisation times (ParamRLS-T). We consider tuning RLS k for a variant of the R function class (R *), where the performance of each parameter value does not change during the run, and for the O M function class, where longer runs favour smaller k. We rigorously prove that ParamRLS-F efficiently tunes RLS k for R * for any κ while ParamRLS-T requires at least quadratic κ. For O M ParamRLS-F identifies k = 1 as optimal with linear κ while ParamRLS-T requires a κ of at least Ω(n log n). For smaller κ ParamRLS-F identifies that k > 1 performs better while ParamRLS-T returns k chosen uniformly at random.

Fast Perturbative Algorithm Configurators

“…The cutoff time κ (the number of iterations for which each configuration is executed for each run in a comparison) varied with the choice of problem class. A fitness-based performance metric was used, as recommended in [10,11], in which the winner of a comparison is the configuration which achieves the highest mean fitness in r runs each lasting κ iterations. In each run, both configurators were initialised uniformly at random.…”

“…. , 9, 10}, where k = 1 is the optimal parameter [10], and the next five largest integers were added until {1, 2, . .…”

“…Note that ParamRLS using this operator is equivalent to ParamHS. We analysed the number of configuration comparisons (that is, calls to the better() procedure present in both ParamRLS and ParamILS) required for the configurators to identify the optimal mutation rate (the optimal value χ in the mutation rate χ/n) for the (1+1) EA optimising Ridge and the (1+1) EA optimising LeadingOnes as in [11] and identifying the optimal neighbourhood size k (the number of bits flipped during mutation) for RLS k optimising OneMax as in [10]. Finally, we considered optimising two parameters of the SAT solver SAPS optimising MAX-SAT [17], searching for one of the five best-performing configurations found during an exhaustive search of the parameter space.…”

“…Recently, the performance of ParamRLS and ParamILS was rigorously analysed for tuning simple single-parameter search heuristics for some standard benchmark problems from the literature. It was proved that they can efficiently tune the neighbourhood size k of the randomised local search algorithm (RLS k ) for Ridge and OneMax [10] and the mutation rate of the simple (1+1) EA for Ridge and LeadingOnes [11]. The analyses, though, also reveal some weaknesses of the search operators used by the two algorithm configurators.…”

“…On the other hand, the mutation operator employed by ParamILS, that changes one parameter value uniformly at random, lacks the ability to efficiently fine-tune the current solution by searching locally around the identified parameter values. Indeed both algorithms require linear expected time in the number of parameter values to identify the optimal configurations for the studied unimodal or approximately unimodal parameter spaces induced by the target algorithms and benchmark functions [10,11].…”