Attention Models in Graphs

Abstract

Graph-structured data arise naturally in many different application domains. By representing data as graphs, we can capture entities (i.e., nodes) as well as their relationships (i.e., edges) with each other. Many useful insights can be derived from graph-structured data as demonstrated by an ever-growing body of work focused on graph mining. However, in the real-world, graphs can be both large -with many complex patterns -and noisy which can pose a problem for effective graph mining. An effective way to deal with this issue is to incorporate "attention" into graph mining solutions. An attention mechanism allows a method to focus on task-relevant parts of the graph, helping it to make better decisions. In this work, we conduct a comprehensive and focused survey of the literature on the emerging field of graph attention models. We introduce three intuitive taxonomies to group existing work. These are based on problem setting (type of input and output), the type of attention mechanism used, and the task (e.g., graph classification, link prediction, etc.). We motivate our taxonomies through detailed examples and use each to survey competing approaches from a unique standpoint. Finally, we highlight several challenges in the area and discuss promising directions for future work.

A Comprehensive Survey on Graph Neural Networks

“…for learning from relational data, reviewing part of GNNs under a unified framework. Lee et al [12] conduct a partial survey of GNNs which apply different attention mechanisms. In summary, existing surveys only include some of the GNNs and examine a limited number of works, thereby missing the most recent development of GNNs.…”

Discovering Strategic Behaviors for Collaborative Content-Production in Social Networks

“…Graph Attention Networks. With recent advancements in graph neural networks [43,46,47] and attention mechanisms [18,39], GAT [40] introduces the attention mechanism in network feature aggregation by implicitly prioritizing node neighbors. Several works attempt to extend GAT to dynamic versions: combining recurrent neural networks with GAT [34]; combined attention on structural neighborhood and temporal dynamics [29] and nodeaware attention for user interaction predictions in real-world dynamic graphs [26].…”

Interoperability and machine-to-machine translation model with mappings to machine learning tasks

“…Conventional architectures like recurrent-and convolutional neural networks are based on the assumption that there is repetitive structure in the input. In contrast, GNNs cannot be based on that assumption because graph data is more irregular in nature, see [32] and [33] for an overview of the field. Several concepts from the image recognition and NLP fields have been adapted to GNNs, like graph convolution [34], graph attention [35] and graph embeddings [36].…”