Transformers for Modeling Physical Systems

Geneva, Nicholas; Zabaras, Nicholas

doi:10.48550/arxiv.2010.03957

Cited by 4 publications

(5 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, applying sequence models to predict high-dimensional systems remains a challenge due to their high memory overhead. Dimensionality reduction techniques, such as CNN autoencoders [33,32,26,22,29,16,11,27], POD [44,48,5,31,18,8,47,10], or Koopman operators [24,9,14] can be used to construct a lowdimensional latent space. The auto-regressive sequence model then operates on these linear (POD modes) or nonlinear (CNNs) latents.…”

Section: Related Workmentioning

confidence: 99%

“…The two trajectories from two different Reynolds numbers are hard to distinguish, which makes it challenging for the learned model to capture parameter variations. Although CNN-based embedding methods [14,50,29] are also nonlinear, they cannot handle irregular geometries with unstructured meshes due to the limitation of classic convolution operations. Using pivotal nodes combined with GNN learning, the proposed model is both flexible in dealing data with irregular meshes and effective in capturing state transitions in the system.…”

Section: Datasetsmentioning

confidence: 99%

See 1 more Smart Citation

Predicting Physics in Mesh-reduced Space with Temporal Attention

Han¹,

Han²,

Pffaf³

et al. 2022

Preprint

View full text Add to dashboard Cite

Graph-based next-step prediction models have recently been very successful in modeling complex high-dimensional physical systems on irregular meshes. However, due to their short temporal attention span, these models suffer from error accumulation and drift. In this paper, we propose a new method that captures long-term dependencies through a transformer-style temporal attention model. We introduce an encoder-decoder structure to summarize features and create a compact mesh representation of the system state, to allow the temporal model to operate on a low-dimensional mesh representations in a memory efficient manner. Our method outperforms a competitive GNN baseline on several complex fluid dynamics prediction tasks, from sonic shocks to vascular flow. We demonstrate stable rollouts without the need for training noise and show perfectly phase-stable predictions even for very long sequences. More broadly, we believe our approach paves the way to bringing the benefits of attention-based sequence models to solving high-dimensional complex physics tasks.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Datasetsmentioning

confidence: 99%

Predicting Physics in Mesh-reduced Space with Temporal Attention

Han¹,

Han²,

Pffaf³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…[16] proposed a Spatio-temporal transformer for 3D human motion modeling by learning the evolution of skeleton joints embeddings through space and time. Also, [23] proposed the use of transformer models for the prediction of dynamical systems representative of physical phenomena. Recently, [24] and [25] applied a Spatio-temporal transformer for video action recognition.…”

Section: Related Workmentioning

confidence: 99%

Is attention to bounding boxes all you need for pedestrian action prediction?

Achaji¹,

Moreau²,

Fouqueray³

et al. 2021

Preprint

View full text Add to dashboard Cite

The human driver is no longer the only one concerned with the complexity of the driving scenarios. Autonomous vehicles (AV) are similarly becoming involved in the process. Nowadays, the development of AV in urban places underpins essential safety concerns for vulnerable road users (VRUs) such as pedestrians. Therefore, to make the roads safer, it is critical to classify and predict their future behavior. In this paper, we present a framework based on multiple variations of the Transformer models to reason attentively about the dynamic evolution of the pedestrians' past trajectory and predict its future actions of crossing or not crossing the street. We proved that using only bounding-boxes as input to our model can outperform the previous state-of-the-art models and reach a prediction accuracy of 91% and an F1-score of 0.83 on the PIE dataset up to two seconds ahead in the future. In addition, we introduced a large-size simulated dataset (CP2A) using CARLA for action prediction. Our model has similarly reached high accuracy (91 %) and F1-score (0.91) on this dataset. Interestingly, we showed that pre-training our Transformer model on the simulated dataset and then fine-tuning it on the real dataset can be very effective for the action prediction task.

show abstract

“…In the past, convolutional neural network architectures have been used to capture the spatial characteristics of complex systems [7]. A temporal dimension to these complex systems was introduced by firstly using an auto-regressive network [5,8] and later by using recurrent neural networks(RNN) where the time dependencies are maintained through the introduction of additional parameters [9,10]. A network that combines the convolutional network and RNN is ConvLSTM [1].…”

Section: Introductionmentioning

confidence: 99%

Sequence to sequence AE-ConvLSTM network for modelling the dynamics of PDE systems

Kakka¹

2022

Preprint

View full text Add to dashboard Cite

The article explains the convolutional LSTM (ConvLSTM) network [1] in detail and introduces an improved auto-encoder version of the ConvLSTM network called AE-ConvLSTM. AE-ConvLSTM is also a sequence to sequence network that can predict long time evolution of a dynamical system by passing hidden states from one encoder to another. The network performed well in predicting the dynamic evolution of unsteady 2-D viscous Burgers when trained using data and, in another case, using governing equation (without data), i.e., physics-constrained. Further, AE-ConvLSTM was used in an effort to predict the time evolution of two unsteady Navier-Stokes problems. These problems have coupled pressure and velocity field having different magnitude order, and these fields evolve in time at a different rate. It was observed that the network could be trained using data, but while training using physics-constrained via governing equations, AE-ConvLSTM fails to train for time evolution.

show abstract

Transformers for Modeling Physical Systems

Cited by 4 publications

References 20 publications

Predicting Physics in Mesh-reduced Space with Temporal Attention

Predicting Physics in Mesh-reduced Space with Temporal Attention

Is attention to bounding boxes all you need for pedestrian action prediction?

Sequence to sequence AE-ConvLSTM network for modelling the dynamics of PDE systems

Contact Info

Product

Resources

About