Using machine learning as a surrogate model for agent-based simulations

Abstract

In this proof-of-concept work, we evaluate the performance of multiple machine-learning methods as surrogate models for use in the analysis of agent-based models (ABMs). Analysing agent-based modelling outputs can be challenging, as the relationships between input parameters can be non-linear or even chaotic even in relatively simple models, and each model run can require significant CPU time. Surrogate modelling, in which a statistical model of the ABM is constructed to facilitate detailed model analyses, has been proposed as an alternative to computationally costly Monte Carlo methods. Here we compare multiple machine-learning methods for ABM surrogate modelling in order to determine the approaches best suited as a surrogate for modelling the complex behaviour of ABMs. Our results suggest that, in most scenarios, artificial neural networks (ANNs) and gradient-boosted trees outperform Gaussian process surrogates, currently the most commonly used method for the surrogate modelling of complex computational models. ANNs produced the most accurate model replications in scenarios with high numbers of model runs, although training times were longer than the other methods. We propose that agent-based modelling would benefit from using machine-learning methods for surrogate modelling, as this can facilitate more robust sensitivity analyses for the models while also reducing CPU time consumption when calibrating and analysing the simulation.

A machine learning accelerated inverse design of underwater acoustic polyurethane coatings

“…Although the primary aim of the surrogate ML model in this research is to significantly reduce the prediction time, black box machine leaning models can suffer from issues of interpretability (Angione et al 2022). To address this issue, we have conducted an extensive parametric study (presented in the supplementary information) and explored the effects of variables to identify the physical mechanisms of sound attenuation.…”

“…The use of DNNs have been reported for accelerated and accurate prediction of the acoustic performance of materials (Ciaburro et al 2021;Jeon et al 2020;Iannace et al 2020;Ciaburro et al 2020). Although these models have achieved considerable success in recent years, they still suffer from issues of interpretability, i.e., lack of understanding of the rationale behind the predictions (Angione et al 2022). However, techniques of interpretation such as sensitivity analysis may be used to explore and extract new insights from the complex physical system (Montavon et al 2018).…”

Bridging the Gap between Mechanistic Biological Models and Machine Learning Surrogates

“…First, it is important to assess whether the mechanistic model is complex enough to invest the time in building a surrogate (Renardy et al, 2018). The design of the protocol for obtaining the training data of the ML surrogate should consider aspects such as stochasticity and whether active learning could bring any additional value (Wang et al, 2019;Angione et al, 2022;Pestourie et al, 2020;Lye et al, 2021). Other aspects such as dimensionality reduction of the inputs and/or outputs of the ML surrogate (Liang et al, 2018a,b;Cai et al, 2021;Lu and Ricciuto, 2019;Nikolopoulos et al, 2022), and parameter sensitivity analysis (Renardy et al, 2018) can help to optimise the performance of the model, but also to unravel some information about the dynamics of the system.…”

“…Surrogate models have been used to approximate stochastic mechanistic models, and it has been shown that if sufficient simulations are run, the distribution of the output of these models is approximately deterministic (Wang et al, 2019). Another approach for building surrogates of stochastic models is to include the random seed which was used for the simulations as an input when training the ML model (Angione et al, 2022).…”

Simulation Intelligence: Towards a New Generation of Scientific Methods

“…In some cases we may be able to build a surrogate model of the ABM simulation (i.e. statistical emulation) for efficient execution and uncertainty quantification [266,267].…”