SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation

Xia, Jun; Wu, Lirong; Chen, Jintao; Hu, Bozhen; Li, Stan Z.

doi:10.1145/3485447.3512156

Cited by 146 publications

(105 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Later, InfoGCL [28] diminishes the mutual information between contrastive parts among views while preserving the task-relevant representation. Beyond augmenting graphs, SimGRACE [15] disturbs the model weights and then learns the invariant high-level representation at the output end to alleviate the design of graph augmentation.…”

Section: Related Workmentioning

confidence: 99%

“…Discussion. From the perspective of information usage for model training, our proposed method is the same as the semi-supervised learning task by recent graph contrastive learning methods [11,12,14,15], which use structure information of all graphs and label information of a subset of all graphs for model training. From the perspective of training strategy, the previous methods first pre-train a model via a contrastive loss and then fine-tune the model for downstream tasks.…”

Section: Label-invariant Augmentationmentioning

confidence: 99%

“…Compared Methods and Implementation. We choose two heuristic self-supervised methods, GAE [26] and Infomax [27], and four recent graph contrastive learning methods, MVGRL [14], GraphCL [11], JOAOv2 [12], and SimGRACE [15], for comparison on semi-supervised graph classification task. GAE performs adjacency matrix reconstruction by using a graph convolutional network (GCN) [31] encoder and a simple inner product decoder.…”

Section: Experiments Settingsmentioning

confidence: 99%

“…Similarly, MVGRL [14] conducts the contrastive multi-view representation learning on both node and graph levels. Beyond the above studies to augment graphs, simGRACE [15] perturbs the model parameters for contrastive learning, which can be regarded as an ensemble of model perturbation or a robust regularization. Although exciting, the above studies point out that the effectiveness of graph contrastive learning heavily hinges on ad-hoc data augmentations, which need to be carefully designed or selected per dataset and request more domain knowledge.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Label-invariant Augmentation for Semi-Supervised Graph Classification

Han¹,

Zhang²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recently, contrastiveness-based augmentation surges a new climax in the computer vision domain, where some operations, including rotation, crop, and flip, combined with dedicated algorithms, dramatically increase the model generalization and robustness. Following this trend, some pioneering attempts employ the similar idea to graph data. Nevertheless, unlike images, it is much more difficult to design reasonable augmentations without changing the nature of graphs. Although exciting, the current graph contrastive learning does not achieve as promising performance as visual contrastive learning. We conjecture the current performance of graph contrastive learning might be limited by the violation of the label-invariant augmentation assumption. In light of this, we propose a label-invariant augmentation for graph-structured data to address this challenge. Different from the node/edge modification and subgraph extraction, we conduct the augmentation in the representation space and generate the augmented samples in the most difficult direction while keeping the label of augmented data the same as the original samples. In the semi-supervised scenario, we demonstrate our proposed method outperforms the classical graph neural network based methods and recent graph contrastive learning on eight benchmark graph-structured data, followed by several in-depth experiments to further explore the label-invariant augmentation in several aspects.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Label-invariant Augmentationmentioning

confidence: 99%

Section: Experiments Settingsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Label-invariant Augmentation for Semi-Supervised Graph Classification

Han¹,

Zhang²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Recent advancements in representation learning have been driven by the SSL paradigm, where the goal is to ensure representations have high similarity between positive views of a sample and high dissimilarity between negative views. Existing SSL frameworks can be broadly categorized based on the mechanism adopted for enforcing this consistency: contrastive learning (CL) frameworks [1,8,7,22,29,31,32], such as GraphCL [22], use the InfoNCE loss; approaches that rely only on positive pairs, such as SimSiam [2] and BGRL [24] use Siamese architectures with stop gradient [2] and asymmetric branches [21] respectively; SpecCL [15] uses a spectral clustering loss (SpecLoss) to enforce consistency; others attempt to directly reduce redundancy between views [3,33]. Despite these differences, all methods rely upon data augmentation to generate positive views, which are assumed to share semantics.…”

Section: Introductionmentioning

confidence: 99%

Analyzing Data-Centric Properties for Graph Contrastive Learning

Trivedi¹,

Lubana²,

Heimann³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent analyses of self-supervised learning (SSL) find the following data-centric properties to be critical for learning good representations: invariance to taskirrelevant semantics, separability of classes in some latent space, and recoverability of labels from augmented samples. However, given their discrete, non-Euclidean nature, graph datasets and graph SSL methods are unlikely to satisfy these properties. This raises the question: how do graph SSL methods, such as contrastive learning (CL), work well? To systematically probe this question, we perform a generalization analysis for CL when using generic graph augmentations (GGAs), with a focus on data-centric properties. Our analysis yields formal insights into the limitations of GGAs and the necessity of task-relevant augmentations. As we empirically show, GGAs do not induce task-relevant invariances on common benchmark datasets, leading to only marginal gains over naive, untrained baselines. Our theory motivates a synthetic data generation process that enables control over task-relevant information and boasts pre-defined optimal augmentations. This flexible benchmark helps us identify yet unrecognized limitations in advanced augmentation techniques (e.g., automated methods). Overall, our work rigorously contextualizes, both empirically and theoretically, the effects of data-centric properties on augmentation strategies and learning paradigms for graph SSL.

show abstract

Key Technology of Automation Control Based on Artificial Intelligence Technology

Wang

2021

Advances in Intelligent Systems and Computing

View full text Add to dashboard Cite

Current state-of-the-art approaches for few-shot action recognition achieve promising performance by conducting frame-level matching on learned visual features. However, they generally suffer from two limitations: i) the matching procedure between local frames tends to be inaccurate due to the lack of guidance to force long-range temporal perception; ii) explicit motion learning is usually ignored, leading to partial information loss. To address these issues, we develop a Motion-augmented Long-short Contrastive Learning (MoLo) method that contains two crucial components, including a long-short contrastive objective and a motion autodecoder. Specifically, the long-short contrastive objective is to endow local frame features with long-form temporal awareness by maximizing their agreement with the global token of videos belonging to the same class. The motion autodecoder is a lightweight architecture to reconstruct pixel motions from the differential features, which explicitly embeds the network with motion dynamics. By this means, MoLo can simultaneously learn long-range temporal context and motion cues for comprehensive few-shot matching. To demonstrate the effectiveness, we evaluate MoLo on five standard benchmarks, and the results show that MoLo favorably outperforms recent advanced methods. The source code is available at https://github. com/alibaba-mmai-research/MoLo.

show abstract

SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation

Cited by 146 publications

References 10 publications

Label-invariant Augmentation for Semi-Supervised Graph Classification

Label-invariant Augmentation for Semi-Supervised Graph Classification

Analyzing Data-Centric Properties for Graph Contrastive Learning

Key Technology of Automation Control Based on Artificial Intelligence Technology

Contact Info

Product

Resources

About