Modeling Localness for Self-Attention Networks

Abstract

Self-attention networks have proven to be of profound value for its strength of capturing global dependencies. In this work, we propose to model localness for self-attention networks, which enhances the ability of capturing useful local context. We cast localness modeling as a learnable Gaussian bias, which indicates the central and scope of the local region to be paid more attention. The bias is then incorporated into the original attention distribution to form a revised distribution. To maintain the strength of capturing long distance dependencies and enhance the ability of capturing shortrange dependencies, we only apply localness modeling to lower layers of self-attention networks. Quantitative and qualitative analyses on Chinese⇒English and English⇒German translation tasks demonstrate the effectiveness and universality of the proposed approach.

Leveraging Local and Global Patterns for Self-Attention Networks

“…We refer readers to Appendix A.1 for the details of our data and experimental settings. Prior studies reveal that modeling locality in lower layers can achieve better performance (Shen et al, 2018;Yu et al, 2018;Yang et al, 2018). Therefore, we merely apply the locality model at the lowest two layers of the encoder.…”

“…Previous work has shown that modeling locality benefits SANs for certain tasks. Luong et al (2015) proposed a Gaussian-based local attention with a predictable position; Sperber et al (2018) differently applied a local method with variable window size for acoustic task; Yang et al (2018) investigated the affect of the dynamical local Gaussian bias by combining these two approaches for the translation task. Different from these methods using a learnable local scope, Yang et al (2019b) and Wu et al (2019) restricted the attention area with fixed size by borrowing the concept of convolution into SANs.…”

“…Comparison to Existing Approaches We reimplement and compare several existing methods (Sperber et al, 2018;Luong et al, 2015;Yang et al, 2018Yang et al, , 2019b upon TRANSFORMER. Table 1 reports the results on the En-De test set.…”

Look Harder: A Neural Machine Translation Model with Hard Attention

“…Model BLEU EN-DE EN-FR Vaswani et al (2017) Transformer big 28.40 41.00 Transformer big + sequence-loss 28.75 41.47 Yang et al (2018) Transformer big + localness 28.89 n/a this work…”

Improving Multi-head Attention with Capsule Networks

“…3 Related Work Attention Mechanism Attention was first introduced in for machine translation tasks by [2] and it already has become an essential part in different architectures [7,13,26] though that they may have different forms. Many works are trying to modify the attention part for different purposes [3,14,16,22,23,25,29]. Our work is mainly related to the work which tries to improve the multi-head attention mechanism in the Transformer model.…”