Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformer

Abstract

A. We introduce a universal class of geometric deep learning models, called metric hypertransformers (MHTs), capable of approximating any adapted map F : X Z → Y Z with approximable complexity, where X ⊆ R d and Y is any suitable metric space, and X Z (resp. Y Z ) capture all discrete-time paths on X (resp. Y ). Suitable spaces Y include various (adapted) Wasserstein spaces, all Fréchet spaces admitting a Schauder basis, and a variety of Riemannian manifolds arising from information geometry. Even in the static case, where f : X → Y is a Hölder map, our results provide the first (quantitative) universal approximation theorem compatible with any such X and Y . Our universal approximation theorems are quantitative, and they depend on the regularity of F, the choice of activation function, the metric entropy and diameter of X , and on the regularity of the compact set of paths whereon the approximation is performed. Our guiding examples originate from mathematical finance. Notably, the MHT models introduced here are able to approximate a broad range of stochastic processes' kernels, including solutions to SDEs, many processes with arbitrarily long memory, and functions mapping sequential data to sequences of forward rate curves.

A Brief Survey on the Approximation Theory for Sequence Modelling

“…We set W i o = 0 in the attention block, so that the input directly goes into the pointwise feed-forward block. From the previous discussion, the pointwise feedforward network can be constructed to give an output y (1) : R → R 2τ+1 , such that y (1) (s) = c s [ φ 0,s (x(s)), . .…”

“…then we have Attn(y (1) )(1) = ∑ τ s=1 y (1) (s). Hence, the final output after the feed-forward network with linear readout c ⊤ = (1, .…”

“…To date, there are few -if any -Jackson or Berstein-type results for sequence modelling using the Transformer. We mention a related series of works on static function approximation with a variant of the Transformer architecture [1,48,49]. Here, the targets are continuous functions H : [0, 1] τ → K, and K ⊂ R n is a compact set.…”