On Calibration of Graph Neural Networks for Node Classification

Liu, Tong; Liu, Yushan; Hildebrandt, Marcel; Joblin, Mitchell; Li, Hang; Tresp, Volker

doi:10.1109/ijcnn55064.2022.9892866

Cited by 3 publications

(6 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The generally reported underconfidence of GCNs 27 , 28 was in line with our results for the GCNConv operator. We observed overconfidence to a higher degree using the GCNConv operator than the GraphConv operator.…”

Section: Discussionsupporting

confidence: 92%

Graph convolutional networks for automated intracranial artery labeling

Vos,

Ruigrok,

Bhat

et al. 2024

J. Med. Imag.

View full text Add to dashboard Cite

.PurposeUnruptured intracranial aneurysms (UIAs) can cause aneurysmal subarachnoid hemorrhage, a severe and often lethal type of stroke. Automated labeling of intracranial arteries can facilitate the identification of risk factors associated with UIAs. This study aims to improve intracranial artery labeling using atlas-based features in graph convolutional networks.ApproachWe included three-dimensional time-of-flight magnetic resonance angiography scans from 150 individuals. Two widely used graph convolutional operators, GCNConv and GraphConv, were employed in models trained to classify 12 bifurcations of interest. Cross-validation was applied to explore the effectiveness of atlas-based features in node classification. The results were tested for statistically significant differences using a Wilcoxon signed-rank test. Model repeatability and calibration were assessed on the test set for both operators. In addition, we evaluated model interpretability and node feature contribution using explainable artificial intelligence.ResultsAtlas-based features led to statistically significant improvements in node classification (p<0.05). The results showed that the best discrimination and calibration performances were obtained using the GraphConv operator, which yielded a mean recall of 0.87, precision of 0.90, and expected calibration error of 0.02.ConclusionsThe addition of atlas-based features improved node classification results. The GraphConv operator, which incorporates higher-order structural information during training, is recommended over the GCNConv operator based on the accuracy and calibration of predicted outcomes.

show abstract

Section: Discussionsupporting

confidence: 92%

Graph convolutional networks for automated intracranial artery labeling

Vos,

Ruigrok,

Bhat

et al. 2024

J. Med. Imag.

View full text Add to dashboard Cite

show abstract

“…Lalitha et al, 40 Rizk et al, 41 and Caldarola et al 42 are some trailblazing literatures that proposed to model clients as nodes in graphs where the collaborative training is analogous to neighborhood aggregation in graph data learning. FedGraphNN and Liu et al 43,44 are prominent benchmark surveys that have contributed to examine the applications and theoretical insights into GNN-based FL across graphs in diverse data domains. However, graph FL encounters unique challenges stemming from graph-specific heterogeneities, such as inconsistent node-and edge-level semantics.…”

Section: Appendix C Related Workmentioning

confidence: 99%

Federated learning for cross-institution brain network analysis

Xie,

Yang,

Cui

et al. 2024

Medical Imaging 2024: Computer-Aided Diagnosis

View full text Add to dashboard Cite

Recent advancements in neuroimaging techniques have sparked a growing interest in understanding the complex interactions between anatomical regions of interest (ROIs), forming into brain networks that play a crucial role in various clinical tasks, such as neural pattern discovery and disorder diagnosis. In recent years, graph neural networks (GNNs) have emerged as powerful tools for analyzing network data. However, due to the complexity of data acquisition and regulatory restrictions, brain network studies remain limited in scale and are often confined to local institutions. These limitations greatly challenge GNN models to capture useful neural circuitry patterns and deliver robust downstream performance. As a distributed machine learning paradigm, federated learning (FL) provides a promising solution in addressing resource limitation and privacy concerns, by enabling collaborative learning across local institutions (i.e., clients) without data sharing. While the data heterogeneity issues have been extensively studied in recent FL literature, cross-institutional brain network analysis presents unique data heterogeneity challenges, that is, the inconsistent ROI parcellation systems and varying predictive neural circuitry patterns across local neuroimaging studies. To this end, we propose FedBrain, a GNN-based personalized FL framework that takes into account the unique properties of brain network data. Specifically, we present a federated atlas mapping mechanism to overcome the feature and structure heterogeneity of brain networks arising from different ROI atlas systems, and a clustering approach guided by clinical prior knowledge to address varying predictive neural circuitry patterns regarding different patient groups, neuroimaging modalities and clinical outcomes. Comparing to existing FL strategies, our approach demonstrates superior and more consistent performance, showcasing its strong potential and generalizability in cross-institutional connectomebased brain imaging analysis.

show abstract

“…While DNNs lean towards overconfidence, GNNs tend to exhibit underconfident. In GNN calibration, there are generally two types of loss functions that regularize the training phase: the graph calibration loss (GCL) (Wang, Yang, and Cheng 2022) and the confidence-reward loss (CRL) (Liu et al 2022). The post-hoc calibration methods in GNNs, such as CaGCN (Wang et al 2021), RBS (Liu et al 2022), and GATS (Hsu et al 2022), train additional networks on the validation set.…”

Section: Gnn Calibrationmentioning

confidence: 99%

“…In GNN calibration, there are generally two types of loss functions that regularize the training phase: the graph calibration loss (GCL) (Wang, Yang, and Cheng 2022) and the confidence-reward loss (CRL) (Liu et al 2022). The post-hoc calibration methods in GNNs, such as CaGCN (Wang et al 2021), RBS (Liu et al 2022), and GATS (Hsu et al 2022), train additional networks on the validation set. They use the GNN output, such as logits, as the input for the calibration model and generate nodespecific temperature coefficients.…”

Section: Gnn Calibrationmentioning

confidence: 99%

See 1 more Smart Citation

Improving GNN Calibration with Discriminative Ability: Insights and Strategies

Fang,

Li,

Chen

et al. 2024

AAAI

View full text Add to dashboard Cite

The widespread adoption of Graph Neural Networks (GNNs) has led to an increasing focus on their reliability. To address the issue of underconfidence in GNNs, various calibration methods have been developed to gain notable reductions in calibration error. However, we observe that existing approaches generally fail to enhance consistently, and in some cases even deteriorate, GNNs' ability to discriminate between correct and incorrect predictions. In this study, we advocate the significance of discriminative ability and the inclusion of relevant evaluation metrics. Our rationale is twofold: 1) Overlooking discriminative ability can inadvertently compromise the overall quality of the model; 2) Leveraging discriminative ability can significantly inform and improve calibration outcomes. Therefore, we thoroughly explore the reasons why existing calibration methods have ineffectiveness and even degradation regarding the discriminative ability of GNNs. Building upon these insights, we conduct GNN calibration experiments across multiple datasets using a straightforward example model, denoted as DC(GNN). Its excellent performance confirms the potential of integrating discriminative ability as a key consideration in the calibration of GNNs, thereby establishing a pathway toward more effective and reliable network calibration.

show abstract

On Calibration of Graph Neural Networks for Node Classification

Cited by 3 publications

References 6 publications

Graph convolutional networks for automated intracranial artery labeling

Graph convolutional networks for automated intracranial artery labeling

Federated learning for cross-institution brain network analysis

Improving GNN Calibration with Discriminative Ability: Insights and Strategies

Contact Info

Product

Resources

About