Rational neural networks

Boullé, Nicolas; Nakatsukasa, Yuji; Townsend, Alex

doi:10.48550/arxiv.2004.01902

Cited by 2 publications

(5 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It can also be observed that the RMSE, MAE, and MAPE indicators of each variable in the PIRNN method were relatively stable in terms of both training and testing set errors. This further indicates that the PIRNN using rational activation functions exhibits extremely robust network fitting ability, which is consistent with the theoretical derivation of rational activation functions discussed in literatures [89][90][91].…”

Section: Numerical Experimentssupporting

confidence: 89%

“…In the NN used in this paper, rational activation functions will be used to replace the traditional activation function, as the NN based on rational activation function have been proved to have superior approximation ability and lower training cost than the NN based on ReLU and its variants via rigorous approximation theory, a large number of theoretical deduction and practical applications [89][90][91].…”

Section: A Rational Activation Functionmentioning

confidence: 99%

See 1 more Smart Citation

Compressible Non-Newtonian Fluid Based Road Traffic Flow Equation Solved by Physical-Informed Rational Neural Network

Yang,

Li,

Nai

et al. 2024

IEEE Access

View full text Add to dashboard Cite

The study of road traffic flow theory utilizes physics and applied mathematics to analyze relevant parameters and their relationships quanlitatively and quantitatively, in order to explore their dynamic changes. The fluid dynamics model used for traffic flow analysis is highly favored by scholars due to its solid mathematical foundation and good simulation results. However, existing models have two main shortcomings: firstly, existing research is mostly limited to non-viscoelastic fluid equation or incompressible non-Newtonian fluid equation, making it difficult to accurately describe the viscosity state and micro cluster properties of the actual traffic flow; secondly, the existing non-Newtonian fluid partial differential equations (PDEs) rely heavily on the finite element method (FEM) for solving, requiring higher computational cost, larger storage space, and more constraint conditions. Thus, in this paper, a traffic flow equation based on compressible non-Newtonian fluid has been constructed, and it has been solved by using physical-informed rational neural network (PIRNN) and noise heavy-ball acceleration gradient descent (NHAGD) to ensure learning and training speed and accuracy. Numerical results indicate that the proposed method can truly reflect the gradual change process in the viscosity of traffic flow, and has better solving performance than traditional FEM and physical-informed neural network (PINN) with activation functions; under the same conditions, the prediction error of the proposed method is also smaller than that of traditional traffic flow models.

show abstract

Section: Numerical Experimentssupporting

confidence: 89%

Section: A Rational Activation Functionmentioning

confidence: 99%

Compressible Non-Newtonian Fluid Based Road Traffic Flow Equation Solved by Physical-Informed Rational Neural Network

Yang,

Li,

Nai

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…The relative L 2 error is presented in Table 4.1 and the training process can be seen in Figure 4.1. Numerical results show that the poly-sine-Gaussian activation has the best performance in terms of the moving-average error and its best historical error is only slightly worse than the rational activation function recently proposed in [5]. We would like to remark that rational activation functions in [5] works well for regression problems but fails in our PDE problems without any meaningful solutions.…”

Section: Discontinuous Function Regressionmentioning

confidence: 87%

“…Numerical results show that the poly-sine-Gaussian activation has the best performance in terms of the moving-average error and its best historical error is only slightly worse than the rational activation function recently proposed in [5]. We would like to remark that rational activation functions in [5] works well for regression problems but fails in our PDE problems without any meaningful solutions. Hence, we only compare reproducing activation functions with rational activation functions in this example.…”

Section: Discontinuous Function Regressionmentioning

confidence: 87%

“…Part I: Scientific Computing Applications. In Part I, we will compare the proposed poly-sine-Gaussian reproducing activation function with existing activation functions, e.g., the ReLU function, the ReLU 3 function, the rational activation function in [5]. We will also provide ablation study to justify the combination of A = {x, x 2 , sin(x), exp(−x 2 )} in the poly-sin-Gaussian network.…”

Section: Note That Bmentioning

confidence: 99%

See 1 more Smart Citation

Reproducing Activation Function for Deep Learning

Liang¹,

Lyu²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper, we propose the reproducing activation function to improve deep learning accuracy for various applications ranging from computer vision problems to scientific computing problems. The idea of reproducing activation functions is to employ several basic functions and their learnable linear combination to construct neuronwise data-driven activation functions for each neuron. Armed with such activation functions, deep neural networks can reproduce traditional approximation tools and, therefore, approximate target functions with a smaller number of parameters than traditional neural networks. In terms of training dynamics of deep learning, reproducing activation functions can generate neural tangent kernels with a better condition number than traditional activation functions lessening the spectral bias of deep learning. As demonstrated by extensive numerical tests, the proposed activation function can facilitate the convergence of deep learning optimization for a solution with higher accuracy than existing deep learning solvers for audio/image/video reconstruction, PDEs, and eigenvalue problems.

show abstract

Rational neural networks

Cited by 2 publications

References 38 publications

Compressible Non-Newtonian Fluid Based Road Traffic Flow Equation Solved by Physical-Informed Rational Neural Network

Compressible Non-Newtonian Fluid Based Road Traffic Flow Equation Solved by Physical-Informed Rational Neural Network

Reproducing Activation Function for Deep Learning

Contact Info

Product

Resources

About