Certified Graph Unlearning

Abstract

Graph-structured data is ubiquitous in practice and often processed using graph neural networks (GNNs). With the adoption of recent laws ensuring the "right to be forgotten", the problem of graph data removal has become of significant importance. To address the problem, we introduce the first known framework for certified graph unlearning of GNNs. In contrast to standard machine unlearning, new analytical and heuristic unlearning challenges arise when dealing with complex graph data. First, three different types of unlearning requests need to be considered, including node feature, edge and node unlearning. Second, to establish provable performance guarantees, one needs to address challenges associated with feature mixing during propagation. The underlying analysis is illustrated on the example of simple graph convolutions (SGC) and their generalized PageRank (GPR) extensions, thereby laying the theoretical foundation for certified unlearning of GNNs. Our empirical studies on six benchmark datasets demonstrate excellent performance-complexity trade-offs when compared to complete retraining methods and approaches that do not leverage graph information. For example, when unlearning 20% of the nodes on the Cora dataset, our approach suffers only a 0.1% loss in test accuracy while offering a 4-fold speed-up compared to complete retraining. Our scheme also outperforms unlearning methods that do not leverage graph information with a 12% increase in test accuracy for a comparable time complexity. Our code will be released upon acceptance. * Equal contribution Preprint. Under review.

Unlearning Nonlinear Graph Classifiers in the Limited Training Data Regime

“…Here, "approximate" refers to the fact that unlearning is not exact (as it would be for completely retrained models) but more akin to the parametrized notion of differential privacy Dwork (2011); Guo et al (2020); Sekhari et al (2021) (see Section 3 for more details). With the adoption of GSTs, we show that our nonlinear framework enables provable data removal (similar results are currently only available for linear models (Guo et al, 2020;Chien et al, 2022)) and provides theoretical unlearning complexity guarantees. These two problems are hard to tackle for deep neural network models like GNNs Xu et al (2019).…”

“…Only a handful of works have taken the initial steps towards machine unlearning of graphs. proposes a sharding-based method for exact graph unlearning, while Chien et al (2022) introduces approximate graph unlearning methods that come with theoretical (certified) guarantees. However, these works only focus on node classification tasks and are in general not directly applicable to graph classification.…”

“…However, these works only focus on node classification tasks and are in general not directly applicable to graph classification. Moreover, the latter method Chien et al (2022) only considers linear model while the work described herein includes nonlinear graph transformations (i.e., GSTs). Note that although DP-GNNs (Daigavane et al, 2021;Sajadmanesh et al, 2022) can also be used to achieve graph unlearning, they not only focus on node classification, but also require a high "privacy cost" to unlearn even one single node or edge without causing significant performance degradation (Chien et al, 2022).…”

“…Moreover, the latter method Chien et al (2022) only considers linear model while the work described herein includes nonlinear graph transformations (i.e., GSTs). Note that although DP-GNNs (Daigavane et al, 2021;Sajadmanesh et al, 2022) can also be used to achieve graph unlearning, they not only focus on node classification, but also require a high "privacy cost" to unlearn even one single node or edge without causing significant performance degradation (Chien et al, 2022). Similar observations regarding the relationship between DP and approximate unlearning were made in the context of unstructured unlearning Guo et al (2020); Sekhari et al (2021).…”

“…However, most state-of-the-art DP-GNNs focus on node classification instead of graph classification tasks, and it remains an open problem to design efficient GNNs for graph classification problem while preserving node-level privacy. Moreover, it has been shown in Chien et al (2022) that DP-GNNs often require a high "privacy cost" ( ≥ 5) (see Equation ( 3)) to unlearn one node without introducing significant negative effects on model performance. In contrast, we find that in practice, our proposed unlearning approach based on GSTs only requires = 1.…”