Rethinking Attention with Performers

Abstract

We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity or low-rankness. To approximate softmax attentionkernels, Performers use a novel Fast Attention Via positive Orthogonal Random features approach (FAVOR+), which may be of independent interest for scalable kernel methods. FAVOR+ can be also used to efficiently model kernelizable attention mechanisms beyond softmax. This representational power is crucial to accurately compare softmax with other kernels for the first time on large-scale tasks, beyond the reach of regular Transformers, and investigate optimal attention-kernels. Performers are linear architectures fully compatible with regular Transformers and with strong theoretical guarantees: unbiased or nearly-unbiased estimation of the attention matrix, uniform convergence and low estimation variance. We tested Performers on a rich set of tasks stretching from pixel-prediction through text models to protein sequence modeling. We demonstrate competitive results with other examined efficient sparse and dense attention methods, showcasing effectiveness of the novel attention-learning paradigm leveraged by Performers.

Learning Operators with Coupled Attention

“…More recently, in the context of transformer architectures, a number of approximations have been proposed to reduce the complexity of such computations to be linear in the number of kernel points O(N ). A non-exhaustive list of references include Linformers [52], Performers [53], Nyströformers [54] and Fast Transformers [55].…”

QuadTree Attention for Vision Transformers

“…Due to the quadratic computational complexity, the computation of full attention is unaffordable when dealing with long sequence tokens. Therefore, many works design efficient transformers, aiming to reduce computational complexity (Katharopoulos et al, 2020;Choromanski et al, 2020;Lee et al, 2019;Ying et al, 2018). Current efficient transformers can be categorized into three classes.…”

“…Current efficient transformers can be categorized into three classes. 1) Linear approximate attention (Katharopoulos et al, 2020;Choromanski et al, 2020;Beltagy et al, 2020;Zaheer et al, 2020) approximates the full attention matrix by linearizing the softmax attention and thus can accelerate the computation by first computing the product of keys and values. 2) Inducing point-based linear transformers (Lee et al, 2019;Ying et al, 2018) use learned inducing points with fixed size to compute attention with input tokens, thus can reduce the computation to linear complexity.…”

GateFormer: Speeding Up News Feed Recommendation with Input Gated Transformers

“…Besides, efficient transformers are proposed, which may reduce the time complexity of self-attention from quadratic to linear (or log-linear). For exam-ple, Linformer and Performer (Choromanski et al, 2020) leverage low-rank selfattention; Sparse Transformers (Child et al, 2019) and Big Bird (Zaheer et al, 2020) utilize sparse self-attention; Reformer introduces learnable attention patterns, and Synthesizer (Tay et al, 2021) introduces randomized attention patterns.…”