p-Power Exponential Mechanisms for Differentially Private Machine Learning

Abstract

Differentially private stochastic gradient descent (DP-SGD) that perturbs the clipped gradients is a popular approach for private machine learning. Gaussian mechanism GM, combined with the moments accountant (MA), has demonstrated a much better privacy-utility tradeoff than using the advanced composition theorem. However, it is unclear whether the tradeoff can be further improved by other mechanisms with different noise distributions. To this end, we extend GM (p = 2) to the generalized p-power exponential mechanism (pEM with p > 0) family and show its privacy guarantee. Straightforwardly, we can enhance the privacy-utility tradeoff of GM by searching noise distribution in the wider mechanism space. To implement pEM in practice, we design an effective sampling method and extend MA to pEM for tightly estimating privacy loss. Besides, we formally prove the non-optimality of GM based on the variation method. Numerical experiments validate the properties of pEM and illustrate a comprehensive comparison between pEM and the other two state-of-the-art methods. Experimental results show that pEM is preferred when the noise variance is relatively small to the signal and the dimension is not too high.

DPBA-WGAN: A Vector-Valued Differential Private Bilateral Alternative Scheme on WGAN for Image Generation

“…Combining with conditional filtering of noise based on an adaptive Gaussian mechanism [3] to prevent excessive noise, achieving the expected utility and privacy. Using the p-Power exponential mechanism(EM) [4] when the noise variance is quite small relative to the signal and the 8 VOLUME XX, 2022 dimension is not too high. Variational Bayesian privacypreserving frameworks based on optimal Bayesian inference methods [5] is another way to solve the high cumulative privacy loss causing by the noise.…”