From block-Toeplitz matrices to differential equations on graphs: towards a general theory for scalable masked Transformers

Choromański, Krzysztof; Lin, Han Chieh; Chen, Haoxian; Zhang, Tianyi; Sehanobish, Arijit; Likhosherstov, Valerii; Parker-Holder, Jack; Sarlós, Tamás; Weller, Adrian; Weingarten, Thomas

doi:10.48550/arxiv.2107.07999

Cited by 1 publication

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the graph domain, linear transformers are not well studied. Choromanski et al [10] are the first to adapt Performer-style attention kernelization to small graphs.…”

Section: Related Workmentioning

confidence: 99%

“…To the best of our knowledge, application of efficient attention models has not yet been thoroughly studied in the graph domain, e.g., only one work [10] explores the adaptation of Performer-style [11] attention approximation on small graphs. Particular challenges emerge with explicit edge features that are incorporated as attention bias in fully-connected graph transformers [34,59].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Recipe for a General, Powerful, Scalable Graph Transformer

Rampášek¹,

Galkin²,

Dwivedi³

et al. 2022

Preprint

View full text Add to dashboard Cite

We propose a recipe on how to build a general, powerful, scalable (GPS) graph Transformer with linear complexity and state-of-the-art results on a diverse set of benchmarks. Graph Transformers (GTs) have gained popularity in the field of graph representation learning with a variety of recent publications but they lack a common foundation about what constitutes a good positional or structural encoding, and what differentiates them. In this paper, we summarize the different types of encodings with a clearer definition and categorize them as being local, global or relative. Further, GTs remain constrained to small graphs with few hundred nodes, and we propose the first architecture with a complexity linear to the number of nodes and edges O(N + E) by decoupling the local real-edge aggregation from the fully-connected Transformer. We argue that this decoupling does not negatively affect the expressivity, with our architecture being a universal function approximator for graphs. Our GPS recipe consists of choosing 3 main ingredients: (i) positional/structural encoding, (ii) local message-passing mechanism, and (iii) global attention mechanism. We build and open-source a modular framework 1 that supports multiple types of encodings and that provides efficiency and scalability both in small and large graphs. We test our architecture on 11 benchmarks and show very competitive results on all of them, show-casing the empirical benefits gained by the modularity and the combination of different strategies.

show abstract

“…In the graph domain, linear transformers are not well studied. Choromanski et al [10] are the first to adapt Performer-style attention kernelization to small graphs.…”

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%