Reinforcement Learning for Efficient Design Space Exploration With Variable Fidelity Analysis Models

Abstract

Reinforcement learning algorithms can autonomously learn to search a design space for high-performance solutions. However, modern engineering often entails the use of computationally-intensive simulation, which can lead to slower design timelines with highly iterative approaches such as RL. This work provides a reinforcement learning framework that leverages models of varying fidelity to enable an effective solution search while reducing overall computational needs. Specifically, it utilizes models of varying fidelity while training the agent, iteratively progressing from low- to high-fidelity. To demonstrate the effectiveness of the proposed framework, we apply it to two multimodal multi-objective constrained mixed integer nonlinear design problems involving the components of a ground and aerial vehicle. Specifically, for each problem we utilize a high-fidelity and a low-fidelity deep neural network surrogate model, trained on performance data generated from underlying ground truth models. A tradeoff between solution quality and the proportion of low-fidelity surrogate model usage is observed. Specifically, high quality solutions are achieved with substantial reductions in computational expense, showcasing the effectiveness of the framework for design problems where the use of just a high-fidelity model is infeasible. This solution quality-computational efficiency tradeoff is contextualized by visualizing the exploration behavior of the design agents.

DS-Viz: a method for visualising design spaces

“…Despite these challenges, many attempts have been made to create design space visualisations. We briefly review here the most notable examples: (a) parallel coordinates plots (Fischer et al, 2014), (b) computationally-derived design spaces (Agrawal and McComb, 2023;Danhaive and Mueller, 2021), (c) set-based design spaces (Nickel et al, 2022) (d) genealogy trees (Bayırlı and Börekçi, 2022;Shah et al, 2003), and (e) conceptual design spaces (Gero and Milovanovic, 2022), which are illustrated in Figure 1.…”

“…In this approach, the design space can be constructed by programmatically changing the values of the variables and observing the resulting outcomes, potentially employing dimensionality reduction techniques to represent individual solutions as points in 2D or 3D space embeddings. Agrawal and McComb (2023) report that such a space can be explored by machine learning agents (see Figure 1(b)) and Danhaive and Mueller (2021) discuss how design landscapes can support parametric design. The core limitation of creating design spaces in this way is related to the design problem formulation that must be amenable to a mathematical parametrisation.…”