The reliability of confidence intervals for computational effort comparisons

Walker, Matthew G.; Edwards, Howard; Messom, Chris

doi:10.1145/1276958.1277294

Cited by 10 publications

(9 citation statements)

References 4 publications

(10 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The lack of an obvious distribution able to describeÊ confirms the previous result reported by Walker et al in [7], who also failed in finding a probability distribution able to modelÊ. We conjecture that there is an underlying random variable associated to the accumulated success probability, and this random variable is modified by several non-lineal operations such us logarithms and the minimum operator, sô E in some sense follows the same distribution but it has been "contaminated" by those operations.…”

Section: Overview Of Performance Measuressupporting

confidence: 86%

“…He identified three sources of variability: the ceiling operator, the estimation error of the accumulated success probability and the minimum operator. Other works investigated how to use some statistical tools with computational effort, mainly confidence intervals [6], [7], toher authors studied the reliability [8], [9] of confidence intervals or proposed alternative performance measures [10].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An empirical study on the accuracy of computational effort in Genetic Programming

Barrero

R-Moreno

Castaño

et al. 2011

2011 IEEE Congress of Evolutionary Computation (CEC)

View full text Add to dashboard Cite

Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. D. F. Barrero, M. D. R-Moreno, B. Castaño, and D. Camacho, "An empirical study on the accuracy of computational effort in Genetic Programming", in IEEE Congress on Evolutionary Computation (CEC), 2011, pp. 1164 - 1171Some commonly used performance measures in Genetic Programming are those defined by John Koza in his first book. These measures, mainly computational effort and number of individuals to be processed, estimate the performance of the algorithm as well as the difficulty of a problem. Although Koza's performance measures have been widely used in the literature, their behaviour is not well known. In this paper we study the accuracy of these measures and advance in the understanding of the factors that influence them. In order to achieve this goal, we report an empirical study that attempts to systematically measure the effects of two variability sources in the estimation of the number of individuals to be processed and the computational effort. The results obtained in those experiments suggests that these measures, in common experimental setups, and under certain circumstances, might have a high relative error.This work was partially supported by the MICYT project ABANT (TIN2010-19872) and Castilla-La Mancha project PEII09- 0266-664

show abstract

Section: Overview Of Performance Measuressupporting

confidence: 86%

Section: Introductionmentioning

confidence: 99%

An empirical study on the accuracy of computational effort in Genetic Programming

Barrero

R-Moreno

Castaño

et al. 2011

2011 IEEE Congress of Evolutionary Computation (CEC)

View full text Add to dashboard Cite

show abstract

“…He identified three sources of variability and provided empirical data that gave some light to the circumstances that reduces the reliability of the computational effort. More research in this area was done by Walker [13,14], who studied how to apply CIs to the calculus of computational effort, and Niehaus [15], who investigated the statistical properties of the computational effort in steady-state algorithms.…”

Section: Related Workmentioning

confidence: 99%

Statistical Distribution of Generation-to-Success in GP: Application to Model Accumulated Success Probability

Barrero

Castaño

R-Moreno

et al. 2011

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Esta es la versión de autor de la comunicación de congreso publicada en: This is an author produced version of a paper published in: Abstract. Many different metrics have been defined in Genetic Programming. Depending on the experiment requirements and objectives, a collection of measures are selected in order to achieve an understanding of the algorithm behaviour. One of the most common metrics is the accumulated success probability, which evaluates the probability of an algorithm to achieve a solution in a certain generation. We propose a model of accumulated success probability composed by two parts, a binomial distribution that models the total number of success, and a lognormal approximation to the generation-to-success, that models the variation of the success probability with the generation.

show abstract

“…Walker et al [45,46] state that computational effort is only a point statistic, with no confidence interval, so any comparisons made with other techniques are inconclusive. However, they do propose an approach for defining a 95% confidence interval for the true computational effort of a technique, using Wilson's method.…”

Section: Confidence Intervalmentioning

confidence: 99%

“…The proportion of successes, p, is defined as p = r/n, where r is the number of successful runs, and n is the total number of runs. The z norm value is set to 1.96, as this was used in [45,46].…”

Section: Confidence Intervalmentioning

confidence: 99%

Parallel evolution using multi-chromosome cartesian genetic programming

Walker

Völk

Smith

et al. 2009

Genet Program Evolvable Mach

View full text Add to dashboard Cite

Parallel and distributed methods for evolutionary algorithms have concentrated on maintaining multiple populations of genotypes, where each genotype in a population encodes a potential solution to the problem. In this paper, we investigate the parallelisation of the genotype itself into a collection of independent chromosomes which can be evaluated in parallel. We call this multi-chromosomal evolution (MCE). We test this approach using Cartesian Genetic Programming and apply MCE to a series of digital circuit design problems to compare the efficacy of MCE with a conventional single chromosome approach (SCE). MCE can be readily used for many digital circuits because they have multiple outputs. In MCE, an independent chromosome is assigned to each output. When we compare MCE with SCE we find that MCE allows us to evolve solutions much faster. In addition, in some cases we were able to evolve solutions with MCE that we unable to with SCE. In a case-study, we investigate how MCE can be applied to to a single objective problem in the domain of image classification, namely, the classification of breast X-rays for cancer. To apply MCE to this problem, we identify regions of interest (RoI) from the mammograms, divide the RoI into a collection of sub-images and use a chromosome to classify each sub-image. This problem allows us to evaluate various evolutionary mutation operators which can pairwise swap chromosomes either randomly or topographically or reuse chromosomes in place of other chromosomes.

show abstract

The reliability of confidence intervals for computational effort comparisons

Cited by 10 publications

References 4 publications

An empirical study on the accuracy of computational effort in Genetic Programming

An empirical study on the accuracy of computational effort in Genetic Programming

Statistical Distribution of Generation-to-Success in GP: Application to Model Accumulated Success Probability

Parallel evolution using multi-chromosome cartesian genetic programming

Contact Info

Product

Resources

About