Robin Hood and Matthew Effects -- Differential Privacy Has Disparate Impact on Synthetic Data

Abstract

Generative models trained using Differential Privacy (DP) are increasingly used to produce and share synthetic data in a privacy-friendly manner. In this paper, we set out to analyze the impact of DP on these models vis-à-vis underrepresented classes and subgroups of data. We do so from two angles: 1) the size of classes and subgroups in the synthetic data, and 2) classification accuracy on them. We also evaluate the effect of various levels of imbalance and privacy budgets.Our experiments, conducted using three state-of-the-art DP models (PrivBayes, DP-WGAN, and PATE-GAN), show that DP results in opposite size distributions in the generated synthetic data. More precisely, it affects the gap between the majority and minority classes and subgroups, either reducing it (a "Robin Hood" effect) or increasing it ("Matthew" effect). However, both of these size shifts lead to similar disparate impacts on a classifier's accuracy, affecting disproportionately more the underrepresented subparts of the data. As a result, we call for caution when analyzing or training a model on synthetic data, or risk treating different subpopulations unevenly, which might also lead to unreliable conclusions.

DP-SGD vs PATE: Which Has Less Disparate Impact on GANs?

“…First, we note that with increased privacy PATE-GAN, again, has much lower variability/spread in terms of size and a smaller drop in terms of recall. We also clearly observe the opposing size effects the two generative models exhibit, similarly to [14] -DP-WGAN makes the classes more uniform, i.e., large classes are reduced, and small classes are increased, while PATE-GAN further enforces the imbalance, large classes become even bigger.…”

“…We experiment with privacy budgets ( ) of 0.5, 5, 15, and infinity ("non-DP"). We measure the class distributions in the resulting synthetic datasets as well as class recall from classifiers (logistic regression similar to [14]) trained on the real/synthetic data and tested on put-aside test data. We also report RMSE for sizes and truncated 2 RMSE (TRMSE) for recall weighted by the real sizes in App.…”

“…Finally, analyzing the DP disparity on generative models, Cheng et al [8] show that training classifiers on balanced DP synthetic images could result in increased majority subgroup influence and utility degradation. Focusing on tabular data, Pereira et al [28] look at single-attribute subgroup fairness and overall classification while Ganev et al [14] analyze class as well as single/multi-attribute subgroup classification parity over a variety of imbalances and privacy budgets. They find that the disparate effects of DP could be opposing depending on the specific generative model and DP mechanism.…”

“…Our work is perhaps closest in spirit to [14,36] but we focus on generative models unlike the former and use image data and have a more disciplined approach to constructing the class imbalances, unlike the latter.…”

“…For example, in the case of deep learning classifiers [4,12,30] empirically illustrate the disparate degradation caused by DP-SGD. However, comparisons between DP-SGD and PATE are still relatively unstudied in this light, with Uniyal et al [36] doing so for classifiers, and more recently, Ganev et al [14] demonstrating the said effects in generative models trained on imbalanced tabular data. To fill this gap, we set out to examine and compare two GAN models trained with DP guarantees (DP-WGAN and PATE-GAN) on imbalanced image data (MNIST) in several imbalance settings.…”