Graph Attention Auto-Encoders

Abstract

Auto-encoders have emerged as a successful framework for unsupervised learning. However, conventional auto-encoders are incapable of utilizing explicit relations in structured data. To take advantage of relations in graph-structured data, several graph autoencoders have recently been proposed, but they neglect to reconstruct either the graph structure or node attributes. In this paper, we present the graph attention auto-encoder (GATE), a neural network architecture for unsupervised representation learning on graph-structured data. Our architecture is able to reconstruct graphstructured inputs, including both node attributes and the graph structure, through stacked encoder/decoder layers equipped with self-attention mechanisms. In the encoder, by considering node attributes as initial node representations, each layer generates new representations of nodes by attending over their neighbors' representations. In the decoder, we attempt to reverse the encoding process to reconstruct node attributes. Moreover, node representations are regularized to reconstruct the graph structure. Our proposed architecture does not need to know the graph structure upfront, and thus it can be applied to inductive learning. Our experiments demonstrate competitive performance on several node classification benchmark datasets for transductive and inductive tasks, even exceeding the performance of supervised learning baselines in most cases.

Mesh-Based Graph Convolutional Neural Networks for Modeling Materials With Microstructure

“…This issue is partially offset by sparse storage of the adjacency matrix, in general, and largely ameliorated by data that is on the same discretization. In future work we will investigate low-rank approximations to the adjacency matrix (Kanada et al, 2018;Lebedev et al, 2014;Richard et al, 2012;Savas and Dhillon, 2011;Tai et al, 2015), dimensionality reduction techniques (Belkin and Niyogi, 2003;He and Niyogi, 2004), and the use of graph autoencoders (Hasanzadeh et al, 2019;Kipf and Welling, 2016b;Liao et al, 2016;Salehi and Davulcu, 2019) to reduce the mesh-based graphs in-line. We are also pursuing the larger topic of processing images with multiresolution filters (Zhang et al, 2018), e.g., spanning the pixel to the cluster level.…”

Mesh-based graph convolutional neural networks for modeling materials with microstructure

“…This issue is partially offset by sparse storage of the adjacency matrix, in general, and largely ameliorated by data that is on the same discretization, as in this work. In future work we will investigate low-rank approximations to the adjacency matrix [80][81][82][83][84], dimensionality reduction techniques [85,86], and the use of graph auto-encoders [87][88][89][90] to reduce the mesh-based graphs in-line. We are also pursuing the larger topic of processing image with multi-resolution filters [91], e.g.…”

Graph autoencoder with constant dimensional latent space

“…We apply two known techniques to boost the autoencoder efficiency (Salehi & Davulcu, 2019). Since indexing of vertices is arbitrary, we proceed as follows: The vertices are sorted by their degree in decreasing order.…”

“…Attention (Vaswani et al, 2017) was introduced to graph VAE by Salehi & Davulcu (2019) as a crucial component of both the encoder and the decoder. Khan & Kleinsteuber (2021) proposed a graph VAE aiming to maximize the similarity between the embeddings of neighboring and more distant vertices while minimizing the redundancy between the components of these embeddings.…”