Confidence Intervals for Computational Effort Comparisons

Walker, Matthew G.; Edwards, Howard; Messom, Chris

doi:10.1007/978-3-540-71605-1_3

Cited by 14 publications

(13 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Given the cumulative probability of success, P (i), of a number of GP runs, we can calculate how many runs would be required, R(i, z), in order to find a solution at generation i with probability z 1 [10]:…”

Section: Computational Effortmentioning

confidence: 99%

“…In an earlier paper [10] we offered three methods for the production of confidence intervals for minimum computational effort and concluded that a method based on Wilson's 'score' method [8] was the best choice: it is fairly easy to calculate, practically always produces a valid confidence interval, and the confidence intervals included the true computational effort an appropriate proportion of the time.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The reliability of confidence intervals for computational effort comparisons

Walker

Edwards

Messom

2007

Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation

Self Cite

View full text Add to dashboard Cite

This paper analyses the reliability of confidence intervals for Koza's computational effort statistic. First, we conclude that dependence between the observed minimum generation and the observed cumulative probability of success leads to the production of more reliable confidence intervals for our preferred method. Second, we show that confidence intervals from 80% to 95% have appropriate levels of performance. Third, simulated data is used to consider the effect of large minimum generations and the confidence intervals are again found to be reliable. Finally, results from four large datasets collected from real genetic programming experiments are used to provide even more empirical evidence that the method for producing confidence intervals is reliable.

show abstract

Section: Computational Effortmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

The reliability of confidence intervals for computational effort comparisons

Walker

Edwards

Messom

2007

Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation

Self Cite

View full text Add to dashboard Cite

show abstract

“…He identified three sources of variability: the ceiling operator, the estimation error of the accumulated success probability and the minimum operator. Other works investigated how to use some statistical tools with computational effort, mainly confidence intervals [6], [7], toher authors studied the reliability [8], [9] of confidence intervals or proposed alternative performance measures [10].…”

Section: Introductionmentioning

confidence: 99%

An empirical study on the accuracy of computational effort in Genetic Programming

Barrero

R-Moreno

Castaño

et al. 2011

2011 IEEE Congress of Evolutionary Computation (CEC)

View full text Add to dashboard Cite

Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. D. F. Barrero, M. D. R-Moreno, B. Castaño, and D. Camacho, "An empirical study on the accuracy of computational effort in Genetic Programming", in IEEE Congress on Evolutionary Computation (CEC), 2011, pp. 1164 - 1171Some commonly used performance measures in Genetic Programming are those defined by John Koza in his first book. These measures, mainly computational effort and number of individuals to be processed, estimate the performance of the algorithm as well as the difficulty of a problem. Although Koza's performance measures have been widely used in the literature, their behaviour is not well known. In this paper we study the accuracy of these measures and advance in the understanding of the factors that influence them. In order to achieve this goal, we report an empirical study that attempts to systematically measure the effects of two variability sources in the estimation of the number of individuals to be processed and the computational effort. The results obtained in those experiments suggests that these measures, in common experimental setups, and under certain circumstances, might have a high relative error.This work was partially supported by the MICYT project ABANT (TIN2010-19872) and Castilla-La Mancha project PEII09- 0266-664

show abstract

“…Another particularity of the 4-parity histogram is the rugosity of its shape, the other histograms are notably smoother. It is because the histograms only show successful runs, and in case of the 4-parity they are much lower (305) than, let say, the artificial ant (13,168).…”

Section: Empirical Model Of Generation-to-successmentioning

confidence: 97%

“…He identified three sources of variability and provided empirical data that gave some light to the circumstances that reduces the reliability of the computational effort. More research in this area was done by Walker [13,14], who studied how to apply CIs to the calculus of computational effort, and Niehaus [15], who investigated the statistical properties of the computational effort in steady-state algorithms.…”

Section: Related Workmentioning

confidence: 99%

Statistical Distribution of Generation-to-Success in GP: Application to Model Accumulated Success Probability

Barrero

Castaño

R-Moreno

et al. 2011

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Esta es la versión de autor de la comunicación de congreso publicada en: This is an author produced version of a paper published in: Abstract. Many different metrics have been defined in Genetic Programming. Depending on the experiment requirements and objectives, a collection of measures are selected in order to achieve an understanding of the algorithm behaviour. One of the most common metrics is the accumulated success probability, which evaluates the probability of an algorithm to achieve a solution in a certain generation. We propose a model of accumulated success probability composed by two parts, a binomial distribution that models the total number of success, and a lognormal approximation to the generation-to-success, that models the variation of the success probability with the generation.

show abstract

Confidence Intervals for Computational Effort Comparisons

Cited by 14 publications

References 3 publications

The reliability of confidence intervals for computational effort comparisons

The reliability of confidence intervals for computational effort comparisons

An empirical study on the accuracy of computational effort in Genetic Programming

Statistical Distribution of Generation-to-Success in GP: Application to Model Accumulated Success Probability

Contact Info

Product

Resources

About