A Continuous-time Stochastic Gradient Descent Method for Continuous Data

Kusaka, Jin; Latz, Jonas; Liu, Chenguang; Schönlieb, Carola‐Bibiane

doi:10.48550/arxiv.2112.03754

Cited by 4 publications

(6 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We follow [52] in using Latin Hybercube Sampling [60], a quasi-random approach for space filling sampling, to obtain the collocation points for training. In our experiments, we re-sample the collocation points in every epoch to get a better coverage of the sampling domain; see also [29]. As mentioned above, the differential operator A and any derivatives in the boundary condition are evaluated using automatic differentiation [4].…”

Section: Physics-informed Neural Networkmentioning

confidence: 99%

Can Physics-Informed Neural Networks beat the Finite Element Method?

Grossmann¹,

Komorowska²,

Latz³

et al. 2023

Preprint

View full text Add to dashboard Cite

Partial differential equations play a fundamental role in the mathematical modelling of many processes and systems in physical, biological and other sciences. To simulate such processes and systems, the solutions of PDEs often need to be approximated numerically. The finite element method, for instance, is a usual standard methodology to do so. The recent success of deep neural networks at various approximation tasks has motivated their use in the numerical solution of PDEs. These so-called physics-informed neural networks and their variants have shown to be able to successfully approximate a large range of partial differential equations. So far, physics-informed neural networks and the finite element method have mainly been studied in isolation of each other. In this work, we compare the methodologies in a systematic computational study. Indeed, we employ both methods to numerically solve various linear and nonlinear partial differential equations: Poisson in 1D, 2D, and 3D, Allen-Cahn in 1D, semilinear Schrödinger in 1D and 2D. We then compare computational costs and approximation accuracies. In terms of solution time and accuracy, physics-informed neural networks have not been able to outperform the finite element method in our study. In some experiments, they were faster at evaluating the solved PDE.

show abstract

Section: Physics-informed Neural Networkmentioning

confidence: 99%

Can Physics-Informed Neural Networks beat the Finite Element Method?

Grossmann¹,

Komorowska²,

Latz³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Stochastic gradient descent is often computationally advantageous compared to normal gradient descent [26] due to computational efficiency as well as being able to escape local minimisers in non-convex optimisation problems [13,34]. As mentioned earlier, the theory employed in this work is based on the continuous-time analysis of stochastic gradient descent by Latz [23] that was further generalised in [21], but is somewhat orthogonal to the diffusion-based continuous-time analysis of SGD of, e.g. [25].…”

Section: Literature Overviewmentioning

confidence: 99%

Subsampling in ensemble Kalman inversion

2023

View full text Add to dashboard Cite

We consider the Ensemble Kalman Inversion which has been recently introduced as an efficient, gradient-free optimization method to estimate unknown parameters in an inverse setting. In the case of large data sets, the Ensemble Kalman Inversion becomes computationally infeasible as the data misfit needs to be evaluated for each particle in each iteration. Here, randomised algorithms like stochastic gradient descent have been demonstrated to successfully overcome this issue by using only a random subset of the data in each iteration, so-called subsampling techniques. Based on a recent analysis of a continuous-time representation of stochastic gradient methods, we propose, analyse, and apply subsampling-techniques within Ensemble Kalman Inversion. Indeed, we propose two different subsampling techniques: either every particle observes the same data subset (single subsampling) or every particle observes a different data subset (batch subsampling).

show abstract

“…In a setting where the differential inclusions are actually differential equations and sufficiently smooth, one can sometimes show that ( x(t)) t≥0 → (x(t)) t≥0 in a weak sense, as λ → 0. We refer to [20,23] for results of this type and a general perspective on stochastic approximation in continuous time.…”

Section: Problem Setting and Motivationmentioning

confidence: 99%

“…Indeed, we aim to show that the stochastic approximations can approximate the deterministic dynamics at any accuracy. Results of this type have been discussed in [20,23] considering smooth stochastic optimisation and in [17] considering Markov chain Monte Carlo. The theoretical foundation is given by Kushner's perturbed test function theory [21,22].…”

Section: Approximation Propertiesmentioning

confidence: 99%

Gradient flows and randomised thresholding: sparse inversion and classification

Latz¹

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Sparse inversion and classification problems are ubiquitous in modern data science and imaging. They are often formulated as non-smooth minimisation problems. In sparse inversion, we minimise, e.g., the sum of a data fidelity term and an L1/LASSO regulariser. In classification, we consider, e.g., the sum of a data fidelity term and a non-smooth Ginzburg-Landau energy. Standard (sub)gradient descent methods have shown to be inefficient when approaching such problems. Splitting techniques are much more useful: here, the target function is partitioned into a sum of two subtarget functions -each of which can be efficiently optimised. Splitting proceeds by performing optimisation steps alternately with respect to each of the two subtarget functions.In this work, we study splitting from a stochastic continuous-time perspective. Indeed, we define a differential inclusion that follows one of the two subtarget function's negative subgradient at each point in time. The choice of the subtarget function is controlled by a binary continuous-time Markov process. The resulting dynamical system is a stochastic approximation of the underlying subgradient flow. We investigate this stochastic approximation for an L1-regularised sparse inversion flow and for a discrete Allen-Cahn equation minimising a Ginzburg-Landau energy. In both cases, we study the longtime behaviour of the stochastic dynamical system and its ability to approximate the underlying subgradient flow at any accuracy. We illustrate our theoretical findings in a simple sparse estimation problem and also in a low-dimensional classification problem.

show abstract

A Continuous-time Stochastic Gradient Descent Method for Continuous Data

Cited by 4 publications

References 26 publications

Can Physics-Informed Neural Networks beat the Finite Element Method?

Can Physics-Informed Neural Networks beat the Finite Element Method?

Subsampling in ensemble Kalman inversion

Gradient flows and randomised thresholding: sparse inversion and classification

Contact Info

Product

Resources

About