Towards Understanding and Refining the General Program Synthesis Benchmark Suite with Genetic Programming

Forstenlechner, Stefan; Fagan, David; Nicolau, Miguel; O’Neill, Michael

doi:10.1109/cec.2018.8477953

Cited by 14 publications

(19 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…and various control flow techniques. These problems have been addressed in several studies, using multiple genetic programming systems using lexicase selection including PushGP [7-10, 12, 14, 19, 20] and grammar guided GP [3][4][5][6], as well as at least one non-evolutionary program synthesis technique [25].…”

Section: Benchmark Problemsmentioning

confidence: 99%

Lexicase selection of specialists

Helmuth

Pantridge²,

Spector

2019

Proceedings of the Genetic and Evolutionary Computation Conference

View full text Add to dashboard Cite

Lexicase parent selection filters the population by considering one random training case at a time, eliminating any individuals with errors for the current case that are worse than the best error in the selection pool, until a single individual remains. This process often stops before considering all training cases, meaning that it will ignore the error values on any cases that were not yet considered. Lexicase selection can therefore select specialist individuals that have poor errors on some training cases, if they have great errors on others and those errors come near the start of the random list of cases used for the parent selection event in question. We hypothesize here that selecting these specialists, which may have poor total error, plays an important role in lexicase selection's observed performance advantages over error-aggregating parent selection methods such as tournament selection, which select specialists much less frequently. We conduct experiments examining this hypothesis, and find that lexicase selection's performance and diversity maintenance degrade when we deprive it of the ability of selecting specialists. These findings help explain the improved performance of lexicase selection compared to tournament selection, and suggest that specialists help drive evolution under lexicase selection toward global solutions.

show abstract

Section: Benchmark Problemsmentioning

confidence: 99%

Lexicase selection of specialists

Helmuth

Pantridge²,

Spector

2019

Proceedings of the Genetic and Evolutionary Computation Conference

View full text Add to dashboard Cite

show abstract

“…However, given more time to evolve, novelty-lexicase selection converged to local optima much less often than lexicase selection, finding solutions throughout all 1000 generations while lexicase selection often failed to find many solutions after 300-500 generations. The lexicase results mirror some extended 600 generation runs on this same benchmark suite using grammar-guided GP, where lexicase selection often plateaued after approximately 300 generations [4]. The ability of novelty-lexicase selection to avoid premature convergence and continue effectively exploring the search space shows great promise whenever evolution requires many generations to construct complex solution programs.…”

Section: Discussionmentioning

confidence: 65%

Comparing and combining lexicase selection and novelty search

Jundt

Helmuth

2019

Proceedings of the Genetic and Evolutionary Computation Conference

View full text Add to dashboard Cite

Lexicase selection and novelty search, two parent selection methods used in evolutionary computation, emphasize exploring widely in the search space more than traditional methods such as tournament selection. However, lexicase selection is not explicitly driven to select for novelty in the population, and novelty search suffers from lack of direction toward a goal, especially in unconstrained, highly-dimensional spaces. We combine the strengths of lexicase selection and novelty search by creating a novelty score for each test case, and adding those novelty scores to the normal error values used in lexicase selection. We use this new novelty-lexicase selection to solve automatic program synthesis problems, and find it significantly outperforms both novelty search and lexicase selection. Additionally, we find that novelty search has very little success in the problem domain of program synthesis. We explore the effects of each of these methods on population diversity and long-term problem solving performance, and give evidence to support the hypothesis that novelty-lexicase selection resists converging to local optima better than lexicase selection.

show abstract

“…boolean, oat) that are used in the problem. This is similar to the grammar of [4]. We call the combination of PSGP with this grammar, variant G Base .…”

Section: Problem Description Knowledge Extractionmentioning

confidence: 96%

“…A watershed moment was the introduction of a program synthesis benchmark suite of 29 problems, systematically selected from sources teaching introductory computer science programming [8]. Four studies are most relevant to this paper: the original benchmarking done using PUSHGP [8], the most recent PUSHGP results using various mutation operators [7], the most recent grammar guided GP eorts [4], and, recently introduced, grammatical evolution with knobelty selection [9]. Henceforth we refer to these as PushGP BM , PushGP MU , G3P, and GE N OV respectively.…”

Section: Program Synthesis With Gpmentioning

confidence: 99%

“…One set of eorts has demonstrated the merits of lexicase selection and used program synthesis to motivate a new mutation operator for PushGP [6,7]. Another has introduced a grammar-guided approach and analyzed the suite's problems to reect upon general grammar design [4,5]. More recently, program synthesis has supported the investigation of how tunable selection-based novelty can help balance search exploration and exploitation.…”

mentioning

confidence: 99%

See 1 more Smart Citation

On domain knowledge and novelty to improve program synthesis performance with grammatical evolution

Hemberg

Kelly

O’Reilly

2019

Proceedings of the Genetic and Evolutionary Computation Conference

View full text Add to dashboard Cite

Programmers solve coding problems with the support of both programming and problem specic knowledge. They integrate this domain knowledge to reason by computational abstraction. Correct and readable code arises from sound abstractions and problem solving. We attempt to transfer insights from such human expertise to genetic programming (GP) for solving automatic program synthesis. We draw upon manual and non-GP Articial Intelligence methods to extract knowledge from synthesis problem denitions to guide the construction of the grammar that Grammatical Evolution uses and to supplement its tness function. We examine the impact of using such knowledge on 21 problems from the GP program synthesis benchmark suite. Additionally, we investigate the compounding impact of this knowledge and novelty search. The resulting approaches exhibit improvements in accuracy on a majority of problems in the eld's benchmark suite of program synthesis problems. CCS CONCEPTS • Computing methodologies → Multi-agent systems;

show abstract

Towards Understanding and Refining the General Program Synthesis Benchmark Suite with Genetic Programming

Cited by 14 publications

References 10 publications

Lexicase selection of specialists

Lexicase selection of specialists

Comparing and combining lexicase selection and novelty search

On domain knowledge and novelty to improve program synthesis performance with grammatical evolution

Contact Info

Product

Resources

About