Improve Transformer Models with Better Relative Position Embeddings

Abstract

Transformer architectures rely on explicit position encodings in order to preserve a notion of word order. In this paper, we argue that existing work does not fully utilize position information. For example, the initial proposal of a sinusoid embedding is fixed and not learnable. In this paper, we first review absolute position embeddings and existing methods for relative position embeddings. We then propose new techniques that encourage increased interaction between query, key and relative position embeddings in the self-attention mechanism. Our most promising approach is a generalization of the absolute position embedding, improving results on SQuAD1.1 compared to previous position embeddings approaches. In addition, we address the inductive property of whether a position embedding can be robust enough to handle long sequences. We demonstrate empirically that our relative position embedding method is reasonably generalized and robust from the inductive perspective. Finally, we show that our proposed method can be adopted as a near drop-in replacement for improving the accuracy of large models with a small computational budget.

Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models

“…When we generated an additional lightweight convolution based on keys, the model performed worse than composite attention alone (GLUE 74.0 compared to 75.2). This result clarifies the findings of Huang et al (2020), who reported only small improvements from query and key-based relative position embeddings for a subset of the GLUE tasks. Grammaticality judgments were particularly sensitive to position information.…”

“…All of our experiments used a convolution kernel size of 17, or eight positions in each direction, a mid-range value that has been found to work well for both relative positions and convolution in language models (Huang et al, 2020;Jiang et al, 2020;Shaw et al, 2018). As in Shaw et al (2018), relative embeddings W C j−i shared weights across heads.…”

“…Referring to the visualization in Figure 1, key-based dynamic convolutions correspond to columns instead of rows. These key-based dynamic lightweight convolutions are the same as the relative embeddings proposed in Huang et al (2020), but they are now formulated as dynamic lightweight convolutions.…”

“…Our work unites and builds upon previous work using convolutions and relative positions in Transformers. We adopted the relative embeddings from Shaw et al (2018) and Huang et al (2020), showing that these embeddings are equivalent to the dynamic lightweight convolutions in Wu et al (2019). Combining these dynamic lightweight convolutions with fixed lightweight convolutions (equivalent to the relative position terms in Raffel et al 2020), we studied relative embeddings under the framework of convolution integrated with selfattention.…”

On the application of BERT models for nanopore methylation detection

“…For any fixed offset k, P E pos+k can be represented as a linear function of P Epos. According to the recent progress (Huang et al, 2020), learnable PE and relative position embedding can help to further improve BERT's performances. Therefore, in the refined BERT model, we use learnable PE and relative position representation.…”