FNetAR: Mixing Tokens with Autoregressive Fourier Transforms

Abstract

In this note we examine the autoregressive generalization of the FNet algorithm, in which selfattention layers from the standard Transformer architecture are substituted with a trivial sparse-uniform sampling procedure based on Fourier transforms. Using the Wikitext-103 benchmark, we demonstrate that FNetAR retains state-of-the-art performance (25.8 ppl) on the task of causal language modeling compared to a Transformer-XL baseline (24.2 ppl) with only half the number self-attention layers, thus providing further evidence for the superfluity of deep neural networks with heavily compounded attention mechanisms. The autoregressive Fourier transform could likely be used for parameter reduction on most Transformer-based time-series prediction models.