GNOT: A General Neural Operator Transformer for Operator Learning

Hao, Zhongkai; Cui, Ying; Wang, Zhengyi; Su, Hang; Dong, Yinpeng; Liu, Songming; Cheng, Ze; Zhu, Jun; Song, Jian

doi:10.48550/arxiv.2302.14376

Cited by 4 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Simultaneously, researchers are exploring the combination of FNO with the attention mechanisms [3] for irregular meshes. This includes the Operator Transformer (OFormer) [39], Mesh-Independent Neural Operator (MINO) [40], and the General Neural Operator Transformer (GNOT) [41]. Besides, the Clifford neural layers [42] use the Clifford algebra to compute multivectors, which provides Clifford-FNO implementations as an extension of FNO.…”

Section: Related Workmentioning

confidence: 99%

A learning-based multiscale method and its application to inelastic impact problems

Liu

Kovachki

et al. 2022

Journal of the Mechanics and Physics of Solids

View full text Add to dashboard Cite

Section: Related Workmentioning

confidence: 99%

A learning-based multiscale method and its application to inelastic impact problems

Liu

Kovachki

et al. 2022

Journal of the Mechanics and Physics of Solids

View full text Add to dashboard Cite

“…(Cao 2021) removes the softmax normalization and introduces Galerkin-type attention to achieve linear scaling. Although efficient transformers (Liu, Xu, and Zhang 2022;Li, Meidani, and Farimani 2022;Hao et al 2023) have been proposed with preserving permutation symmetries (Lee 2022), the demand for flexibility and scalability is still not met for practical use in the real world. Inducing-Point Methods.…”

Section: Related Workmentioning

confidence: 99%

Inducing Point Operator Transformer: A Flexible and Scalable Architecture for Solving PDEs

Lee,

2024

AAAI

View full text Add to dashboard Cite

Solving partial differential equations (PDEs) by learning the solution operators has emerged as an attractive alternative to traditional numerical methods. However, implementing such architectures presents two main challenges: flexibility in handling irregular and arbitrary input and output formats and scalability to large discretizations. Most existing architectures are limited by their desired structure or infeasible to scale large inputs and outputs. To address these issues, we introduce an attention-based model called an inducing point operator transformer (IPOT). Inspired by inducing points methods, IPOT is designed to handle any input function and output query while capturing global interactions in a computationally efficient way. By detaching the inputs/outputs discretizations from the processor with a smaller latent bottleneck, IPOT offers flexibility in processing arbitrary discretizations and scales linearly with the size of inputs/outputs. Our experimental results demonstrate that IPOT achieves strong performances with manageable computational complexity on an extensive range of PDE benchmarks and real-world weather forecasting scenarios, compared to state-of-the-art methods. Our code is publicly available at https://github.com/7tl7qns7ch/IPOT.

show abstract

“…Its success has been extended to other areas, including computer vision tasks [21] and biology [22]. It has also inspired a wide array of scientific applications, in particular PDEs modeling [23][24][25][26][27][28][29][30]. Kovachki et al [16] proposes a kernel integral interpretation of attention.…”

Section: Introductionmentioning

confidence: 99%

“…Learning operators with coupled attention [32] uses attention weights to learn correlations in the output domain and enables sample-efficient training of the model. General neural operator transformer for operator learning [25] proposes a heterogeneous attention architecture that stacks multiple cross-attention layers and uses a geometric gating mechanism to adaptively aggregate features from query points. Additionally, encoding physics-informed inductive biases has also been of great interest because it allows incorporatation of additional system knowledge, making the learning task easier.…”

Section: Introductionmentioning

confidence: 99%

Physics informed token transformer for solving partial differential equations

Lorsung,

Li,

Barati Farimani

2024

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

Solving Partial Differential Equations (PDEs) is the core of many fields of science and engineering. While classical approaches are often prohibitively slow, machine learning models often fail to incorporate complete system information. Over the past few years, transformers have had a significant impact on the field of Artificial Intelligence and have seen increased usage in PDE applications. However, despite their success, transformers currently lack integration with physics and reasoning. This study aims to address this issue by introducing PITT: Physics Informed Token Transformer. The purpose of PITT is to incorporate the knowledge of physics by embedding partial differential equations (PDEs) into the learning process. PITT uses an equation tokenization method to learn an analytically-driven numerical update operator. By tokenizing PDEs and embedding partial derivatives, the transformer models become aware of the underlying knowledge behind physical processes. To demonstrate this, PITT is tested on challenging 1D and 2D PDE neural operator prediction tasks. The results show that PITT outperforms popular neural operator models and has the ability to extract physically relevant information from governing equations.

show abstract

GNOT: A General Neural Operator Transformer for Operator Learning

Cited by 4 publications

References 0 publications

A learning-based multiscale method and its application to inelastic impact problems

A learning-based multiscale method and its application to inelastic impact problems

Inducing Point Operator Transformer: A Flexible and Scalable Architecture for Solving PDEs

Physics informed token transformer for solving partial differential equations

Contact Info

Product

Resources

About