Computational characteristics of feedforward neural networks for solving a stiff differential equation

Abstract

Feedforward neural networks offer a possible approach for solving differential equations. However, the reliability and accuracy of the approximation still represent delicate issues that are not fully resolved in the current literature. Computational approaches are in general highly dependent on a variety of computational parameters as well as on the choice of optimisation methods, a point that has to be seen together with the structure of the cost function. The intention of this paper is to make a step towards resolving these open issues. To this end, we study here the solution of a simple but fundamental stiff ordinary differential equation modelling a damped system. We consider two computational approaches for solving differential equations by neural forms. These are the classic but still actual method of trial solutions defining the cost function, and a recent direct construction of the cost function related to the trial solution method. Let us note that the settings we study can easily be applied more generally, including solution of partial differential equations. By a very detailed computational study, we show that it is possible to identify preferable choices to be made for parameters and methods. We also illuminate some interesting effects that are observable in the neural network simulations. Overall we extend the current literature in the field by showing what can be done in order to obtain useful and accurate results by the neural network approach. By doing this we illustrate the importance of a careful choice of the computational setup.

Adaptive neural-domain refinement for solving time-dependent differential equations

“…The NF approach comes with plenty of parameters. We have already shown in a computational study [26], that they are not independent of each other. Changing one parameter may require another parameter to be changed as well in order to improve or maintain the reliability.…”

“…Also, when there is an unfavourable local minimum close to the initialisation, the optimiser may only find this one. However, fine tuning all the incorporated computational parameters is a challenging task since some of these are not independent of each other [9,26].…”

“…Parameters marked with ( * ) will be separately discussed in the corresponding paragraph. The initial weight values, the SCNF order as well as the number of epochs and training points (n TP ) have been previously investigated and are fixed to suitable values, see [5,26] for further details. Nonetheless, each parameter has its impact on the solution.…”

“…Problem statement and contribution. In our previous work, we investigated the computational characteristics of a small feedforward neural network with only one hidden layer [26]. It is well known that the parameters within a neural network are not independent of each other.…”

Exploring Types of Photonic Neural Networks for Imaging and Computing—A Review

“…Computational methodologies are generally heavily reliant on various computational parameters and the selection of optimization techniques, which must be considered in conjunction with the structure of the cost function. In [ 48 ], the resolution of a straightforward yet pivotal stiff ordinary differential equation representing a damped system is proposed. Two computational strategies are proposed for resolving differential equations using neural forms: the conventional but still relevant approach of trial solutions defining the cost function and a recent direct formulation of the cost function associated with the trial solution process.…”