Invariance-Preserving Localized Activation Functions for Graph Neural Networks

Abstract

Graph signals are signals with an irregular structure that can be described by a graph. Graph neural networks (GNNs) are information processing architectures tailored to these graph signals and made of stacked layers that compose graph convolutional filters with nonlinear activation functions. Graph convolutions endow GNNs with invariance to permutations of the graph nodes' labels. In this paper, we consider the design of trainable nonlinear activation functions that take into consideration the structure of the graph. This is accomplished by using graph median filters and graph max filters, which mimic linear graph convolutions and are shown to retain the permutation invariance of GNNs. We also discuss modifications to the backpropagation algorithm necessary to train local activation functions. The advantages of localized activation function architectures are demonstrated in four numerical experiments: source localization on synthetic graphs, authorship attribution of 19th century novels, movie recommender systems and scientific article classification. In all cases, localized activation functions are shown to improve model capacity.

Towards Finite-Time Consensus with Graph Convolutional Neural Networks

“…The last fully connected layer leaves unaffected the distributed implementation since it is local over the nodes. In this work, we study the effect of three activation functions for distributed consensus: the pointwise ReLU, the pointwise kernel [22], and the local max [23].…”

“…The ReLU term nonlinearizes also the node features. In [23], the authors extended (7) to a neighborhood of order K. This choice, however, is not distributable and we shall not discuss it further. The above activation functions leave unaffected the communication and computational costs of the GCNN, which remain governed by the cost of running all graph filters [cf.…”

Stability of Graph Neural Networks to Relative Perturbations

“…are local activation functions involving neighboring exchanges that also preserve the permutation equivariance property [14].…”

“…Furthermore, graph convolutions are used to build graph neural networks (GNNs), as a cascade of layers each of which applies a graph convolution, followed by a pointwise nonlinearity [9][10][11]. GNNs offer a nonlinear transformation of the input data that has achieved remarkable performance in wireless networks [12], decentralized control of robot swarms [13] and recommendation systems [14], among others [15,16]. Graph filters and GNNs rely heavily on the knowledge of the GSO.…”

The Graphon Fourier Transform

“…For instance, when designing information processing architectures on networks that are bound to grow, we want to avoid making adjustments every time a new node is added to the network. This is the case, for instance, of streaming services that get thousands of new users every day and whose recommendation algorithms run on user-similarity networks [3,4]. Another example is reproducing a certain type of low-dimensional feature analysis on multiple instances of the same type of graph, eg., quantifying air pollution dispersion spectra on air quality sensor networks in different cities (cf.…”