Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View

Abstract

The Transformer architecture is widely used in natural language processing. Despite its success, the design principle of the Transformer remains elusive. In this paper, we provide a novel perspective towards understanding the architecture: we show that the Transformer can be mathematically interpreted as a numerical Ordinary Differential Equation (ODE) solver for a convection-diffusion equation in a multi-particle dynamic system. In particular, how words in a sentence are abstracted into contexts by passing through the layers of the Transformer can be interpreted as approximating multiple particles' movement in the space using the Lie-Trotter splitting scheme and the Euler's method. Given this ODE's perspective, the rich literature of numerical analysis can be brought to guide us in designing effective structures beyond the Transformer. As an example, we propose to replace the Lie-Trotter splitting scheme by the Strang-Marchuk splitting scheme, a scheme that is more commonly used and with much lower local truncation errors. The Strang-Marchuk splitting scheme suggests that the self-attention and position-wise feed-forward network (FFN) sub-layers should not be treated equally. Instead, in each layer, two position-wise FFN sub-layers should be used, and the self-attention sub-layer is placed in between. This leads to a brand new architecture. Such an FFN-attention-FFN layer is "Macaron-like", and thus we call the network with this new architecture the Macaron Net. Through extensive experiments, we show that the Macaron Net is superior to the Transformer on both supervised and unsupervised learning tasks. The reproducible codes and pretrained models can be found at https://github.com/zhuohan123/macaron-net

Darts-Conformer: Towards Efficient Gradient-Based Neural Architecture Search For End-to-End ASR

“…Conformer [16] achieves state-of-theart results on LibriSpeech, outperforming the previous best published Transformer Transducer by 15% relative improvement on the test-other dataset. The core methods they proposed are replacing the original feed-forward layer in the Transformer block into two half-step feed-forward layers which is inspired by Macaron-Net [28], and using a convolution module which contains a gating mechanism after multi-headed self-attention.…”

On the Regularity of Attention

“…We were mathematically inspired by Del Moral (2004) who studied self-interacting "Feynman-Kac models" using semigroup techniques (including contractions for nonlinear operators on measures). An interacting particle interpretation of attention is studied in Lu et al (2019) using tools from dynamical systems theory.…”

IOT: Instance-wise Layer Reordering for Transformer Structures

“…Specific to Transformer structures, various modifications have been proposed. For example, human knowledge powered designs include DynamicConv , Macaron Network (Lu et al, 2019), Reformer (Kitaev et al, 2020) and others (Fonollosa et al, 2019;Ahmed et al, 2017;Shaw et al, 2018). As for automatic searching, neural architecture We show two ordered encoders/decoders here.…”

“…A Transformer model is stacked by several identical blocks, and each block consists of sequentially ordered layers: the self-attention (SA), encoder-decoder attention (ED) (decoder only) and feed-forward (FF) layer. Recently, various modifications have been proposed, where the focus is on replacing or inserting some components (e.g., attention layer/layer norm/position encoding) in standard Transformer Lu et al, 2019;Shaw et al, 2018;So et al, 2019;Ahmed et al, 2017). Despite these Transformer alternatives have achieved improved performances, one critical element is almost neglected in current models, which is how to arrange the components within a Transformer network, i.e., the layer order also matters.…”