Tensorized Transformer for Dynamical Systems Modeling

Abstract

The identification of nonlinear dynamics from observations is essential for the alignment of the theoretical ideas and experimental data. The last, in turn, is often corrupted by the side effects and noise of different natures, so probabilistic approaches could give a more general picture of the process. At the same time, high-dimensional probabilities modeling is a challenging and data-intensive task. In this paper, we establish a parallel between the dynamical systems modeling and language modeling tasks. We propose a transformer-based model that incorporates geometrical properties of the data and provide an iterative training algorithm allowing the fine-grid approximation of the conditional probabilities of high-dimensional dynamical systems.Preprint. Under review.

Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers

“…While our model provides a novel point of view for building representations of collective and individual dynamics, we note that there are also limitations: 1) treating individual systems separately, limits the capacity of individuals (also discussed in [56]) as each individual has limited dimensionality as in the final representation space. 2) our model incorporates individual dynamics in a deterministic way, while many individuals can actually be expressed with a probabilistic model (such as in [57]).…”

“…Additionally, the individual module of EIT models individual dynamics in a deterministic way, but in certain cases (e.g. multi-armed bandit tasks for neural activities), it might be more appropriate to model dynamics and interactions in a probabilistic manner (such as in [70]). • Reducing the need for labels through self-supervised training: While the individual representations of EIT generalize reasonably well across animals in the neural activity experiments, the model currently relies on labels to guide this functional alignment.…”

“…Additionally, the individual module of models individual dynamics in a deterministic way, but in certain cases (e.g. multi-armed bandit tasks for neural activities), it might be more appropriate to model dynamics and interactions in a probabilistic manner (such as in [70]).…”

Transformers for Modeling Physical Systems

“…This work deviates from this pre-existing literature by investigating the use of transformers for the prediction of physical systems, relying entirely on self-attention to model dynamics. In the recent work of Shalova & Oseledets (2020), transformers were tested to predict several low dimensional dynamical ODEs. However, this was entirely focused on learning a single ODE solution and the methods proposed constrained the model to predict low-dimensional systems.…”