RoFormer: Enhanced Transformer with Rotary Position Embedding

Su, Jianlin; Lu, Yu; Pan, Shengfeng; Wen, Bo; Liu, Yunfeng

doi:10.48550/arxiv.2104.09864

Cited by 109 publications

(131 citation statements)

References 15 publications

Supporting

Mentioning

100

Contrasting

Order By: Relevance

“…We replace relative attention in vanilla layers by LSH attention (Kitaev, Kaiser, and Levskaya 2020), which allows us to handle 12288-long sequences. To achieve relative attention parametrization, the LSH attention is combined with rotary positional embeddings (Su et al 2021) layers. In this setup, we reach a score of 3.443 bpd with a (3@1, 12@3, 3@1) architecture which has a total linear cost of 10.…”

Section: Imagenet64mentioning

confidence: 99%

“…Instead of the segment-level recurrence mechanism proposed in that paper, we use shortening to make our model more efficient and feasible to train on longer sequences. Another recently proposed relative attention parametrization is RoFormer (Su et al 2021) where rotary positional embeddings are introduced. We find this work particularly relevant because rotary positional embeddings are compatible with any attention type including efficient attention and can be combined with our model (Section 3.3).…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Hierarchical Transformers Are More Efficient Language Models

Nawrot¹,

Tworkowski²,

Tyrolski³

et al. 2021

Preprint

View full text Add to dashboard Cite

Transformer models yield impressive results on many NLP and sequence modeling tasks. Remarkably, Transformers can handle long sequences which allows them to produce long coherent outputs: full paragraphs produced by GPT-3 or wellstructured images produced by DALL-E. These large language models are impressive but also very inefficient and costly, which limits their applications and accessibility. We postulate that having an explicit hierarchical architecture is the key to Transformers that efficiently handle long sequences. To verify this claim, we first study different ways to downsample and upsample activations in Transformers so as to make them hierarchical. We use the best performing upsampling and downsampling layers to create Hourglassa hierarchical Transformer language model. Hourglass improves upon the Transformer baseline given the same amount of computation and can yield the same results as Transformers more efficiently. In particular, Hourglass sets new stateof-the-art for Transformer models on the ImageNet32 generation task and improves language modeling efficiency on the widely studied enwik8 benchmark. * Equal contribution. Order determined by coin toss.Preprint. Under review.

show abstract

Section: Imagenet64mentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Hierarchical Transformers Are More Efficient Language Models

Nawrot¹,

Tworkowski²,

Tyrolski³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…We use the Rotary positional encoding proposed in [61] and extend it to the 3D case. Given a 3D point S i = (x, y, z) ∈ R 3 , and the it's feature…”

Section: Relative 3d Positional Encodingmentioning

confidence: 99%

Lepard: Learning partial point cloud matching in rigid and deformable scenes

Li¹,

Harada²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…• DEIT-LA, a DEIT model equipped with linearized attention ( §2.2) instead of softmax attention. We also include several variants that improve DEIT-LA, such as PERMUTEFORMER (Chen, 2021), SPE (Liutkus et al, 2021) and Rotary positional embeddings (ROPE, Su et al, 2021) that incorporates relative positional encodings.…”

Section: Algorithm 1 Dynamic Programming For Ripple Attentionmentioning

confidence: 99%

Ripple Attention for Visual Perception with Sub-quadratic Complexity

Lin¹,

Pan²,

Kong³

2021

Preprint

View full text Add to dashboard Cite

Transformer architectures are now central to modeling in natural language processing tasks. At its heart is the attention mechanism, which enables effective modeling of long-term dependencies in a sequence. Recently, transformers have been successfully applied in the computer vision domain, where 2D images are first segmented into patches and then treated as 1D sequences. Such linearization, however, impairs the notion of spatial locality in images, which bears important visual clues. To bridge the gap, we propose ripple attention, a sub-quadratic attention mechanism for visual perception. In ripple attention, contributions of different tokens to a query are weighted with respect to their relative spatial distances in the 2D space. To favor correlations with vicinal tokens yet permit long-term dependencies, we derive the spatial weights through a stick-breaking transformation. We further design a dynamic programming algorithm that computes weighted contributions for all queries in linear observed time, taking advantage of the summed-area table and recent advances in linearized attention. Extensive experiments and analyses demonstrate the effectiveness of ripple attention on various visual tasks.

show abstract

RoFormer: Enhanced Transformer with Rotary Position Embedding

Cited by 109 publications

References 15 publications

Hierarchical Transformers Are More Efficient Language Models

Hierarchical Transformers Are More Efficient Language Models

Lepard: Learning partial point cloud matching in rigid and deformable scenes

Ripple Attention for Visual Perception with Sub-quadratic Complexity

Contact Info

Product

Resources

About