DPNAS: Neural Architecture Search for Deep Learning with Differential Privacy

Abstract

Training deep neural networks (DNNs) for meaningful differential privacy (DP) guarantees severely degrades model utility. In this paper, we demonstrate that the architecture of DNNs has a significant impact on model utility in the context of private deep learning, whereas its effect is largely unexplored in previous studies. In light of this missing, we propose the very first framework that employs neural architecture search to automatic model design for private deep learning, dubbed as DPNAS. To integrate private learning with architecture search, a DP-aware approach is introduced for training candidate models composed on a delicately defined novel search space. We empirically certify the effectiveness of the proposed framework. The searched model DPNASNet achieves state-of-the-art privacy/utility trade-offs, e.g., for the privacy budget of (epsilon, delta)=(3, 1e-5), our model obtains test accuracy of 98.57% on MNIST, 88.09% on FashionMNIST, and 68.33% on CIFAR-10. Furthermore, by studying the generated architectures, we provide several intriguing findings of designing private-learning-friendly DNNs, which can shed new light on model design for deep learning with differential privacy.

Preserving Differential Privacy in Deep Learning Based on Feature Relevance Region Segmentation

“…Wu et al [15] propose an Adaptive Differentially Private Stochastic Gradient Descent (ADPSGD) algorithm, which adjusts the random noise added to the gradient by adaptive step size. Combining private learning with architectural search, Cheng et al [16] propose the DPNASNet model, which achieves a state-of-the-art privacy/utility trade-off.…”

“…To verify the effectiveness of the AFRRS mechanism, the experiments were designed to compare TSA [10], BC [11], DPL-GGC [12], DP-PSAC [13], AUTO clipping [14], ADPSGD [15], DPNASNet [16], and the deep learning model without differential privacy protection (No DP). The results are shown in Table 3 and Table 4.…”

“…Algorithm Accuracy (%) PB ( ) No DP 99.00 -TSA [10] 98.10 2.93 DPL-GGC [12] 98.23 2.85 DP-PSAC [13] 98.24 3.00 AUTO clipping [14] 98.15 3.00 DPNASNet [16] 98.57 3.00 AFRRS 98.80 0.60…”

“…The cause lies in the fact the more privacy budget means the weaker privacy guarantee, and less noise is injected into the model. [10] 66.20 7.53 BC [11] 74.00 3.64 DPL-GGC [12] 67.11 3.19 ADPSGD [15] 69.63 6.40 DPNASNet [16] 68.33 3.00 AFRRS 76.00 2.00…”

Differential Privacy Meets Neural Network Pruning

“…Furthermore, [12] suggested modifying the loss function to promote a fast convergence in DP-SGD. [2] developed a paradigm for neural network architecture search with differential privacy constraints, while [14] a gradient hard thresholding framework that provides good utility guarantees. [8] proposed a grouped gradient clipping mechanism to modulate the gradient weights.…”