Comparison of conceptually different multi-objective Bayesian optimization methods for material design problems

Would you rather search for a point inside of a line or a line inside of a rectangle? This is a type of solution degeneracy that often exists physics-based simulations and wetlab experiments, but constraining these degeneracies is often unsupported or difficult-to-implement in many optimization packages, requiring additional time and expertise. So, is the increase in efficiency worth the cost of implementation? We demonstrate that the compactness of a search space (to what extent degenerate solutions and nonsolutions are removed) can have a significant effect on Bayesian optimization search efficiency via the Ax platform. As our optimization task, we use a physics-based particle packing simulation with seven to nine tunable parameters, depending on the search space compactness, that represent three truncated, discrete log-normal distributions of particle sizes. This physics-based simulation exhibits three qualitatively different degeneracy types: size-invariance, compositional-invariance, and permutation-invariance. The degeneracies are reflected in the outcomes being identical when: 1. all particle sizes are multiplied by a constant factor, 2. the fractional prevalences of the particle types sum to unity, and 3. sets of log-normal distribution parameters are swapped with each other, respectively. This simulation provides fertile ground for assessing the impact of multiple constraints on search efficiency, with a total of eight search space types which ranges from none up to all three constraints imposed simultaneously. Contrary to intuition, we find that, on average, the most compact search space performs worse than the least compact search space ((0.692 ± 0.036) vs. (0.699 ± 0.016) , respectively) over 50 iterations due to the interactions of the non-linear size-invariance degeneracy with other degeneracy types. The most efficient search space in terms of both predicted and validated outcomes is the combination of the composition and permutation constraints, resulting in a mean packing fraction of (0.728±0.010) over 50 iterations, where randomly sampled volume fractions are typically no less than 0.6. We recommend that optimization practitioners in the physical sciences carefully consider the impact of removing search space degeneracies on search efficiency prior to running expensive optimization campaigns.

show abstract

“…• How do these findings compare to other optimization algorithms (e.g. genetic algorithms, random forest based BO [42])?…”

Section: Future Workmentioning

confidence: 99%

Effect of reducible and irreducible search space representations on adaptive design efficiency: a case study on maximizing packing fraction for solid rocket fuel propellant simulations

Baird

Hall

Sparks

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…• How do these findings compare to other optimization algorithms (e.g. random search, genetic algorithms, random forest based BO [57])?…”

Section: Future Workmentioning

confidence: 99%

The most compact search space is not always the most efficient: A case study on maximizing solid rocket fuel packing fraction via constrained Bayesian optimization

Baird

Hall

Sparks

2022

Preprint

View full text Add to dashboard Cite

Would you rather search for a line inside a cube or a point inside a square? This type of solution degeneracy often exists in physics-based simulations and wet-lab experiments, but constraining these degeneracies is often unsupported or difficult to implement in many optimization packages, requiring additional time and expertise. So, are the possible improvements in efficiency worth the cost of implementation? We demonstrate that the compactness of a search space (to what extent and how degenerate solutions and non-solutions are removed) can significantly affect Bayesian optimization search efficiency via the Ax platform. We use a physics-based particle packing simulation with seven to nine tunable parameters, depending on the search space compactness, that represent three truncated, discrete log-normal distributions of particle sizes. This physics-based simulation exhibits three qualitatively different degeneracy types: size-invariance, compositional-invariance, and permutation-invariance. We assess a total of eight search space types which range from none up to all three constraint types imposed simultaneously. We find that leaving the search space unconstrained leads to a large variance in the outcome and that on average, the most constrained search space is not always the most efficient. Likewise, the least constrained search space is not always the least efficient. We recommend that optimization practitioners in the physical sciences carefully consider the impact of removing search space degeneracies on search efficiency before running expensive optimization campaigns.

show abstract

“…Aside from being a performance metric to compare optimisation strategies, HV can also be directly evaluated to guide convergence of various algorithms. Hanaoka et al showed that scalarization-based MOBOs may be best suited for clear exploitation and/or preferential optimisation trajectory of objectives, whereas HV-based MOBOs are better for exploration of the entire search space [75]. Indeed, HV-based approaches empirically show a preference in proposed solutions towards the extrema of a PF [76], [77], and thus can better showcase extrapolation.…”

Section: Hypervolumementioning

confidence: 99%

“…An unavoidable issue of empirically benchmarking optimisation strategies on real-world problems is that some surrogate model must be used in-lieu of a black-box where new data is experimentally validated. Alternatively, a candidate selection problem can be used where optimisation is limited to only proposing new candidates from a pre-labelled dataset until eventually the 'pool' of samples is exhausted [65], [75], [99], [100]. The benefit of this method over surrogate-based methods is that only real data from the black-box is used, rather than data extrapolated from a model approximating its behaviour.…”

Section: B Real-world Benchmarksmentioning

confidence: 99%

Bayesian vs Evolutionary Optimisation in Exploring Pareto Fronts for Materials Discovery

Low¹,

Vissol-Gaudin²,

Lim³

et al. 2022

Preprint

View full text Add to dashboard Cite

<p> With advancements in automated experimental setups, material optimisation and discovery can scale to higher throughput with larger evaluation budgets. Two state-of-the-art algorithms with conceptually different multi-objective optimisation strategies (Bayesian and Evolutionary) are compared on synthetic and real-world datasets. Our results show that the Bayesian optimisation strategy, q-Noisy Expected Hypervolume Improvement (qNEHVI) is superior in finding solutions at the Pareto Front rapidly, and when considering hypervolume improvement as a performance indicator. On the other hand, the Evolutionary optimisation strategy, Unified Non-dominated Sorting Genetic Algorithm III (U-NSGA-III), can exploit the Pareto Front and propose a larger pool of optimal solutions, given sufficient evaluation budget, and thus may be a better choice for materials discovery problems where knowing the complete Pareto Front provides greater scientific value to understanding materials space. We discuss the limitations of using hypervolume as a performance indicator for optimisation strategies, alongside hypervolume-based strategies such as qNEHVI, which do not adequately explain the number of solutions at or near the Pareto Front. We also performed a comparison of both optimisation strategies at different batch sizes to consider throughput capabilities. </p>

show abstract

Comparison of conceptually different multi-objective Bayesian optimization methods for material design problems

Cited by 23 publications

References 31 publications

Effect of reducible and irreducible search space representations on adaptive design efficiency: a case study on maximizing packing fraction for solid rocket fuel propellant simulations

Effect of reducible and irreducible search space representations on adaptive design efficiency: a case study on maximizing packing fraction for solid rocket fuel propellant simulations

The most compact search space is not always the most efficient: A case study on maximizing solid rocket fuel packing fraction via constrained Bayesian optimization

Bayesian vs Evolutionary Optimisation in Exploring Pareto Fronts for Materials Discovery

Contact Info

Product

Resources

About