Permutation Invariant Graph Generation via Score-Based Generative Modeling

Abstract

Learning generative models for graphstructured data is challenging because graphs are discrete, combinatorial, and the underlying data distribution is invariant to the ordering of nodes. However, most of the existing generative models for graphs are not invariant to the chosen ordering, which might lead to an undesirable bias in the learned distribution. To address this difficulty, we propose a permutation invariant approach to modeling graphs, using the recent framework of score-based generative modeling. In particular, we design a permutation equivariant, multi-channel graph neural network to model the gradient of the data distribution at the input graph (a.k.a., the score function). This permutation equivariant model of gradients implicitly defines a permutation invariant distribution for graphs. We train this graph neural network with score matching and sample from it with annealed Langevin dynamics. In our experiments, we first demonstrate the capacity of this new architecture in learning discrete graph algorithms. For graph generation, we find that our learning approach achieves better or comparable results to existing models on benchmark datasets.

Structured Denoising Diffusion Models in Discrete State-Spaces

“…While most works have considered continuous diffusion models, discrete diffusion-like models were described in [43] and applied to text generation and image segmentation data in [20]. Some works [31,29] have dealt with discrete data by embedding it in continuous space and leveraging Gaussian diffusion, but have not applied this to text. Seff et al [42] also considered generation of discrete structured objects using a diffusion-like Markov corruption process.…”

TD-GEN: Graph Generation With Tree Decomposition

“…GraphAF is an autoregressive graph generation model which combines the advantages of sequential models and normalizing flows for high capacity density estimation [2]. GNF and EDP-GNN both use Graph Neural Networks (GNN) for permutation invariant generation of graphs [16,17]. GNF uses reversible GNNs based on normalizing flows, and EDP-GNN uses score matching for the graph generation.…”

Adversarial Stein Training for Graph Energy Models

“…Our approach consists of three components. First using an Edgewise Dense Prediction Graph Neural Network (EDPGNN) [27] architecture we write a powerful permutation invariant energy function. Second we use a novel approach (labeled Adversarial Stein Training) to learn such energy models without requiring computationally burdensome sampling during training.…”

“…We first obtain the number of nodes N in the graph. For this the approach of Ziegler and Rush [41], Niu et al [27] is taken which samples from the empirical multinomial distribution of node sizes in the training data. Once N is fixed, we can sample matrix A ∈ R N ×N via Langevin dynamics on the energy function E θ .…”

“…In this work we utilize GNNs to construct a permutation invariant EBM for distribution over graphs. More specifically we used the learnable multi-channel approach used in [27] to design our energy function. Then we rely on using modification of Stein's Identity [34] to adversarially train EBMs without the need to sample which allows scaling to larger datasets.…”