Deep Constraint-Based Propagation in Graph Neural Networks

Tiezzi, Matteo; Marra, Giuseppe; Melacci, Stefano; Maggini, Marco

doi:10.1109/tpami.2021.3073504

Cited by 11 publications

(5 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The delay is zero for h = D since, in this case, the last stage computes the forward, backward (and update) operations related to a given input at the same time. However, it is not only a matter of changing the stage input, since a similar consideration holds for the values of weights, that get updated (thus they change) at each computational step, as we already anticipated in (7). Even if this will play a role in the evaluation of the gradients, a small learning rate can mitigate abrupt changes in the values of the weights, making the resulting approximation more appropriate.…”

Section: Scalable and Parallel Local Computations Over Timementioning

confidence: 97%

PARTIME: Scalable and Parallel Processing Over Time with Deep Neural Networks

Meloni¹,

Faggi²,

Marullo³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we present PARTIME, a software library written in Python and based on PyTorch, designed specifically to speed up neural networks whenever data is continuously streamed over time, for both learning and inference.Existing libraries are designed to exploit data-level parallelism, assuming that samples are batched, a condition that is not naturally met in applications that are based on streamed data. Differently, PARTIME starts processing each data sample at the time in which it becomes available from the stream. PARTIME wraps the code that implements a feed-forward multi-layer network and it distributes the layer-wise processing among multiple devices, such as Graphics Processing Units (GPUs). Thanks to its pipeline-based computational scheme, PARTIME allows the devices to perform computations in parallel. At inference time this results in scaling capabilities that are theoretically linear with respect to the number of devices. During the learning stage, PARTIME can leverage the non-i.i.d. nature of the streamed data with samples that are smoothly evolving over time for efficient gradient computations. Experiments are performed in order to empirically compare PARTIME with classic non-parallel neural computations in online learning, distributing operations on up to 8 NVIDIA GPUs, showing significant speedups that are almost linear in the number of devices, mitigating the impact of the data transfer overhead.

show abstract

Section: Scalable and Parallel Local Computations Over Timementioning

confidence: 97%

PARTIME: Scalable and Parallel Processing Over Time with Deep Neural Networks

Meloni¹,

Faggi²,

Marullo³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Sampling-based techniques have been developed for fast and scalable GNN training, such as GraphSAGE (Hamilton, Ying, and Leskovec 2017) and FastGCN (Chen, Ma, and Xiao 2018). Simplifying methods (Tiezzi et al 2021;Wu et al 2019) are also proposed to make the GNN models more easily implementable and more efficient. Mixhop (Abu-El-Haija et al 2019) and Graph Diffusion Convolution(GDC) (Klicpera, Weißenberger, and Günnemann 2019) explored combining feature information from multihop neighborhoods, while PPNP (Klicpera, Bojchevski, and Günnemann 2018) and PPRGo (Bojchevski et al 2020) derived an improved propagation scheme of high-order information based on personalized PageRank.…”

Section: Related Workmentioning

confidence: 99%

Graph Pointer Neural Networks

Yang

Wang

Yue

et al. 2022

AAAI

View full text Add to dashboard Cite

Graph Neural Networks (GNNs) have shown advantages in various graph-based applications. Most existing GNNs assume strong homophily of graph structure and apply permutation-invariant local aggregation of neighbors to learn a representation for each node. However, they fail to generalize to heterophilic graphs, where most neighboring nodes have different labels or features, and the relevant nodes are distant. Few recent studies attempt to address this problem by combining multiple hops of hidden representations of central nodes (i.e., multi-hop-based approaches) or sorting the neighboring nodes based on attention scores (i.e., ranking-based approaches). As a result, these approaches have some apparent limitations. On the one hand, multi-hop-based approaches do not explicitly distinguish relevant nodes from a large number of multi-hop neighborhoods, leading to a severe over-smoothing problem. On the other hand, ranking-based models do not joint-optimize node ranking with end tasks and result in sub-optimal solutions. In this work, we present Graph Pointer Neural Networks (GPNN) to tackle the challenges mentioned above. We leverage a pointer network to select the most relevant nodes from a large amount of multi-hop neighborhoods, which constructs an ordered sequence according to the relationship with the central node. 1D convolution is then applied to extract high-level features from the node sequence. The pointer-network-based ranker in GPNN is joint-optimized with other parts in an end-to-end manner. Extensive experiments are conducted on six public node classification datasets with heterophilic graphs. The results show that GPNN significantly improves the classification performance of state-of-the-art methods. In addition, analyses also reveal the privilege of the proposed GPNN in filtering out irrelevant neighbors and reducing over-smoothing.

show abstract

“…Ciano et al [34] proposed a mixed inductive-transductive GNN model, study its properties and introduce an experimental strategy that help to understand and distinguish the role of inductive and transductive learning. Tiezzi et al [35] proposed an approach to learning in GNNs based on constrained optimization in the Lagrangian framework. Learning both the transition function and the node states is the outcome of a joint process, in which the state convergence procedure is implicitly expressed by a constraint satisfaction mechanism, avoiding iterative epoch-wise procedures and the network unfolding.…”

Section: Graph Neural Networkmentioning

confidence: 99%

GMSS: Graph-Based Multi-Task Self-Supervised Learning for EEG Emotion Recognition

Li¹,

Chen²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Previous electroencephalogram (EEG) emotion recognition relies on single-task learning, which may lead to overfitting and learned emotion features lacking generalization. In this paper, a graph-based multi-task self-supervised learning model (GMSS) for EEG emotion recognition is proposed. GMSS has the ability to learn more general representations by integrating multiple self-supervised tasks, including spatial and frequency jigsaw puzzle tasks, and contrastive learning tasks. By learning from multiple tasks simultaneously, GMSS can find a representation that captures all of the tasks thereby decreasing the chance of overfitting on the original task, i.e., emotion recognition task. In particular, the spatial jigsaw puzzle task aims to capture the intrinsic spatial relationships of different brain regions. Considering the importance of frequency information in EEG emotional signals, the goal of the frequency jigsaw puzzle task is to explore the crucial frequency bands for EEG emotion recognition. To further regularize the learned features and encourage the network to learn inherent representations, contrastive learning task is adopted in this work by mapping the transformed data into a common feature space. The performance of the proposed GMSS is compared with several popular unsupervised and supervised methods. Experiments on SEED, SEED-IV, and MPED datasets show that the proposed model has remarkable advantages in learning more discriminative and general features for EEG emotional signals.

show abstract

Deep Constraint-Based Propagation in Graph Neural Networks

Cited by 11 publications

References 21 publications

PARTIME: Scalable and Parallel Processing Over Time with Deep Neural Networks

PARTIME: Scalable and Parallel Processing Over Time with Deep Neural Networks

Graph Pointer Neural Networks

GMSS: Graph-Based Multi-Task Self-Supervised Learning for EEG Emotion Recognition

Contact Info

Product

Resources

About