An empirical study on the accuracy of computational effort in Genetic Programming

Abstract

Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. D. F. Barrero, M. D. R-Moreno, B. Castaño, and D. Camacho, "An empirical study on the accuracy of computational effort in Genetic Programming", in IEEE Congress on Evolutionary Computation (CEC), 2011, pp. 1164 - 1171Some commonly used performance measures in Genetic Programming are those defined by John Koza in his first book. These measures, mainly computational effort and number of individuals to be processed, estimate the performance of the algorithm as well as the difficulty of a problem. Although Koza's performance measures have been widely used in the literature, their behaviour is not well known. In this paper we study the accuracy of these measures and advance in the understanding of the factors that influence them. In order to achieve this goal, we report an empirical study that attempts to systematically measure the effects of two variability sources in the estimation of the number of individuals to be processed and the computational effort. The results obtained in those experiments suggests that these measures, in common experimental setups, and under certain circumstances, might have a high relative error.This work was partially supported by the MICYT project ABANT (TIN2010-19872) and Castilla-La Mancha project PEII09- 0266-664

Better GP benchmarks: community survey results and proposals

“…More comments and detail are available online. 2 The feedback from this discussion is similar to that found in the survey responses mentioned above. There was a general sentiment that things must be improved, and that new benchmarks and better empirical practice were the two most promising ways of making this improvement.…”

“…The most common approach [28] defines the computational effort measure as an estimate of the minimum number of individuals to be processed in a generational algorithm in order to achieve a high probability of discovering a solution. This measure has received significant criticism [37,3,31,34,2,24]. Critics have argued that ideal solution counts are really a measure of how well a method solves trivial problems, and that other measures would be better, such as best fitness of run (appropriate for problems where the goal is optimization) and generalization measures such as final testing against a withheld generalization set, or K fold validation (appropriate for problems where the goal is to perform model-fitting).…”

Genetic programming needs better benchmarks

“…This measure has received significant criticism [51,7,43,47,4]. Critics have noted that ideal solution counts are really a measure of how well a method solves trivial problems, rather than the nontrivial ones found in real world applications.…”