Inductive Biases and Variable Creation in Self-Attention Mechanisms

Abstract

Self-attention, an architectural motif designed to model long-range interactions in sequential data, has driven numerous recent breakthroughs in natural language processing and beyond. This work provides a theoretical analysis of the inductive biases of self-attention modules, where our focus is to rigorously establish which functions and long-range dependencies self-attention blocks prefer to represent. Our main result shows that bounded-norm Transformer layers create sparse variables: they can represent sparse functions of the input sequence, with sample complexity scaling only logarithmically with the context length. Furthermore, we propose new experimental protocols to support this analysis and to guide the practice of training Transformers, built around the large body of work on provably learning sparse Boolean functions.

Learning the Structure of Hub Network Based on Graph Model

“…Our work focuses on the analysis of Maximum Likelihood Estimate (MLE) with transformer function class, which is not covered by previous works. Our bounds are sharper than that ofEdelman et al (2021) on the channel number dependency.…”

“…Following this line,Liao et al (2020) ,Ledent et al (2021) andLin and Zhang (2019) built the generalization bound for graph neural networks and convolutional neural network. These results respected the underlying graph structure and the translation-invariance in the networks Edelman et al (2021). established the generalization bound for transformer, but this result did not reflect the permutation-invariance, still depending on the channel number.…”

“…Wei et al (2022a) presented the approximation and generalization bounds for learning boolean circuits and Turing machines with transformers. Edelman et al (2021) and Li et al (2023) derived the generalization error bound of transformers. In our work, we analyze transformers from both the analytic and statistical sides.…”