Graph Neural Networks for Natural Language Processing: A Survey

Wu, Lingfei; Chen, Yu; Shen, Kai; Guo, Xinwen; Gao, Hanning; Li, Shucheng; Pei, Jian; Long, Bo

doi:10.1561/2200000096

Cited by 85 publications

(21 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Graph neural networks and state space models have emerged as significant areas of interest in the field of natural language processing (NLP). Graph neural networks (GNNs), for instance, have been increasingly employed to capture the relational structure inherent in language data (Wu et al, 2023). They excel in tasks where linguistic elements are interrelated, e.g., in dependency parsing or semantic role labeling (Ji et al, 2019).…”

Section: Related Workmentioning

confidence: 99%

N-Net: A Convolutional Neural Network for Medical Image Segmentation

Dumitru

Peteleaza

Craciun

2023

Preprint

View full text Add to dashboard Cite

This paper presents a novel supervised convolutional neural network architecture, "N-Net", capable of effectively learning and generalizing from small amounts of medical images to perform accurate segmentation tasks. Our model utilizes an encoder-decoder structure with a residual downsampling mechanism and a custom convolutional block to capture and process image information at multiple resolutions in the encoder segment. We employ data augmentation techniques to enrich the training set, thus increasing our model's performance. While our architecture is versatile and applicable to various segmentation tasks, in this study, we demonstrate its capabilities specifically for polyp segmentation in colonoscopy images. We evaluate the performance of our method on several popular benchmark datasets for polyp segmentation, Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB, and ETIS-LARIBPOLYPDB showing that it achieves state-of-the-art results in terms of mean Dice coefficient, Jaccard index, Precision, Recall, and Accuracy. Our approach demonstrates strong generalization capabilities, achieving excellent performance even with limited training data.

show abstract

Section: Related Workmentioning

confidence: 99%

N-Net: A Convolutional Neural Network for Medical Image Segmentation

Dumitru

Peteleaza

Craciun

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Additionally, many scholars have combed and summarized GNNs from different perspectives (such as methods, applications, etc.). For details, please refer to the review [100][101][102][103][104][105][106][107][108][109][110]. Due to its high degree of freedom, good computability, and high reasoning efficiency, the spatial-based method has been widely concerned and developed.…”

Section: Output Layermentioning

confidence: 99%

Graph-based Knowledge Distillation: A survey and experimental evaluation

Liu¹,

Tongya²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Graph data, such as citation networks, social networks, and transportation networks, are prevalent in the real world. Graph neural networks (GNNs) have gained widespread attention for their robust expressiveness and exceptional performance in various graph analysis applications. However, the efficacy of GNNs is heavily reliant on sufficient data labels and complex network models, with the former being challenging to obtain and the latter requiring expensive computational resources. To address the labeled data scarcity and high complexity of GNNs, Knowledge Distillation (KD) has been introduced to enhance existing GNNs. This technique involves transferring the soft-label supervision of the large teacher model to the small student model while maintaining prediction performance. Transferring the KD technique to graph data and graph-based knowledge is a major challenge. This survey offers a comprehensive overview of Graph-based Knowledge Distillation methods, systematically categorizing and summarizing them while discussing their limitations and future directions. This paper first introduces the background of graph and KD. It then provides a comprehensive summary of three types of Graph-based Knowledge Distillation methods, namely Graph-based Knowledge Distillation for deep neural networks (DKD), Graph-based Knowledge Distillation for GNNs (GKD), and Self-Knowledge Distillation based Graph-based Knowledge Distillation (SKD). Each type of method is further divided into knowledge distillation methods based on the output layer, middle layer, and constructed graph. Subsequently, various graph-based knowledge distillation algorithms' ideas are analyzed and compared, concluding with the advantages and disadvantages of each algorithm supported by experimental results. In addition, the applications of graph-based knowledge distillation in computer vision, natural language processing, recommendation systems, and other fields are listed. Finally, the development of graph-based knowledge distillation is summarized and prospectively discussed. We have also released related resources at https://github.com/liujing1023/Graph-based-Knowledge-Distillation.

show abstract

“…GNNs extend the principles of deep learning to graph data, transforming the way in which we handle complex interrelated data and providing breakthroughs in various applications, such as social networks [15][16][17], natural language processing [18][19][20][21], computer vision [22][23][24][25] and recommendation [26][27][28]. The introduction of GNNs addressed these challenges by introducing a method that can learn directly from graph data, preserving and leveraging the inherent relational information.…”

Section: Related Workmentioning

confidence: 99%

Efficient Non-Sampling Graph Neural Networks

et al. 2023

Information

View full text Add to dashboard Cite

A graph is a widely used and effective data structure in many applications; it describes the relationships among nodes or entities. Currently, most semi-supervised or unsupervised graph neural network models are trained based on a very basic operation called negative sampling. Usually, the purpose of the learning objective is to maximize the similarity between neighboring nodes while minimizing the similarity between nodes that are not close to each other. Negative sampling can reduce the time complexity by sampling a small fraction of the negative nodes instead of using all of the negative nodes when optimizing the objective. However, sampling of the negative nodes may fail to deliver stable model performance due to the uncertainty in the sampling procedure. To avoid such disadvantages, we provide an efficient Non-Sampling Graph Neural Network (NS-GNN) framework. The main idea is to use all the negative samples when optimizing the learning objective to avoid the sampling process. Of course, directly using all of the negative samples may cause a large increase in the model training time. To mitigate this problem, we rearrange the origin loss function into a linear form and take advantage of meticulous mathematical derivation to reduce the complexity of the loss function. Experiments on benchmark datasets show that our framework can provide better efficiency at the same level of prediction accuracy compared with existing negative sampling-based models.

show abstract

Graph Neural Networks for Natural Language Processing: A Survey

Cited by 85 publications

References 0 publications

N-Net: A Convolutional Neural Network for Medical Image Segmentation

N-Net: A Convolutional Neural Network for Medical Image Segmentation

Graph-based Knowledge Distillation: A survey and experimental evaluation

Efficient Non-Sampling Graph Neural Networks

Contact Info

Product

Resources

About