Mini-batch optimization enables training of ODE models on large-scale datasets

Abstract

Quantitative dynamic models are widely used to study cellular signal processing. A critical step in modelling is the estimation of unknown model parameters from experimental data. As model sizes and datasets are steadily growing, established parameter optimization approaches for mechanistic models become computationally extremely challenging. Mini-batch optimization methods, as employed in deep learning, have better scaling properties. In this work, we adapt, apply, and benchmark mini-batch optimization for ordinary differential equation (ODE) models, thereby establishing a direct link between dynamic modelling and machine learning. On our main application example, a large-scale model of cancer signaling, we benchmark mini-batch optimization against established methods, achieving better optimization results and reducing computation by more than an order of magnitude. We expect that our work will serve as a first step towards mini-batch optimization tailored to ODE models and enable modelling of even larger and more complex systems than what is currently possible.

Network inference from perturbation time course data

“…Increasing the network size will quadratically increase the number of unknown parameters, which will significantly increase the computational requirements for obtaining robust solutions. Yet, recent work has shown that large estimation problems in ODE models may be broken into several smaller problems 75 , which may be applicable here, and is likely to yield large computational speed up by allowing parallelization of much smaller tasks. However, theory on how to merge potentially discrepant results between independently estimated overlapping subnetworks would need to be derived.…”

The Synergy between Deep Learning and Organs-on-Chips for High-Throughput Drug Screening: A Review

“…To that end, four strategies are widely utilized during training. (i) Mini-batch [ 25 , 26 ]: mini-batch only utilizes a batch of data instead of full data during each update to reduce memory usage and improve the training efficiency. (ii) Stochastic gradient descent (SGD) [ 27 , 28 ]: The SGD strategy adds random factors in gradient calculation, which is generally fast and benefits the model’s generalization.…”

Efficient Bayesian inference for mechanistic modelling with high-throughput data

“…A common practice in data science is to use the elbow method [34], which was first employed by Thorndike [35]. The elbow method amounts to choosing K as the elbow of the curve of the minimum of the objective function in Eq (15) as K is varied. The interpretation of picking the elbow of the curve, in clustering, corresponds to choosing K such that adding futher clusters does not provide a significantly better fit to the data.…”

“…Inspired by the method of stochastic gradient descent (SGD) [12][13][14] in machine learning, we propose a minibatch approach to tackle this issue: for each comparison between simulated and observed data, we use a stochastically sampled subset (minibatch) of the data. A similar minibatch method has been employed very recently by Stapor et al [15] to successfully calibrate ordinary differential equation (ODE) models with a significant improvement in computational performance, and by Seita et al [16] within the context of MCMC, likewise with a significant computational speed-up. We demonstrate that choosing a large enough minibatch ensures that the relevant signatures in the observed data can be accurately estimated, while avoiding unnecessary comparisons that slow down inference.…”