Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers

Abstract

Transformer models have achieved state-of-the-art results across a diverse range of domains. However, concern over the cost of training the attention mechanism to learn complex dependencies between distant inputs continues to grow. In response, solutions that exploit the structure and sparsity of the learned attention matrix have blossomed. However, real-world applications that involve long sequences, such as biological sequence analysis, may fall short of meeting these assumptions, precluding exploration of these models. To address this challenge, we present a new Transformer architecture, Performer, based on Fast Attention Via Orthogonal Random features (FAVOR). Our mechanism scales linearly rather than quadratically in the number of tokens in the sequence, is characterized by sub-quadratic space complexity and does not incorporate any sparsity pattern priors. Furthermore, it provides strong theoretical guarantees: unbiased estimation of the attention matrix and uniform convergence. It is also backwards-compatible with pre-trained regular Transformers. We demonstrate its effectiveness on the challenging task of protein sequence modeling and provide detailed theoretical analysis.

A Survey of Transformers

“…2 ), 𝑙 = 1, 𝑓 1 = exp and thus guarantees unbiased and nonnegative approximation of dot-product attention. This approach is more stable than Choromanski et al [18] and reports better approximation results.…”

“…Performer [18,19] uses random feature maps that approximate the scoring function of Transformer. The random feature maps take functions 𝑓 1 , • • • , 𝑓 𝑙 : R → R and ℎ : R 𝐷 → R.…”

“…The first version of Performer [18] is inspired from the random Fourier feature map [105] that was originally used to approximate Gaussian kernel. It uses trigonometric functions with…”

What's Hidden in a One-layer Randomly Weighted Transformer?

“…In the early days of neural networks, fixed random layers (Baum, 1988;Schmidt et al, 1992;Pao et al, 1994) have been studied in reservoir computing (Maass et al, 2002;Jaeger, 2003;Lukoševičius and Jaeger, 2009), "random kitchen sink" kernel machines Recht, 2008, 2009), and so on. Recently, random features have also been extensively explored for modern neural networks in deep reservoir computing networks (Scardapane and Wang, 2017;Gallicchio and Micheli, 2017;Shen et al, 2021), random kernel feature (Peng et al, 2021;Choromanski et al, 2020), and applications in text classification (Conneau et al, 2017;Wieting and Kiela, 2019), summarization (Pilault et al, 2020) and probing (Voita and Titov, 2020). Compressing Transformer.…”

FNetAR: Mixing Tokens with Autoregressive Fourier Transforms

“…The success of the Transformer has proven that compounding these SEM's results in a uniquely effective function approximator for even the most complex correlation functions, such as those that determine the structure of natural languages. However, there is also a growing body of evidence [9][10][11][12][13][14][15] that many of these computations are superfluous and that many state-of-the-art results can be reproduced with significantly fewer learnable parameters, making computations more efficient and generally leading to faster training and better performing models Optimizing the Transformer is currently an active field of research, and currently many of the most effective methods involve complicated rearrangements of traditional architectures. In a recent work [16], the authors presented a uniquely simplified variation on the standard autoencoding Transformer architecture, in which they substitute several self-attention sublayers with a computationally trivial procedure for mixing tokens using Fourier transform coefficients, thus benefiting from the machinery of FFT algorithms such as Cooley-Tukey.…”