Individual Privacy Accounting via a Renyi Filter

Feldman, Vitaly; Zrnic, Tijana

doi:10.48550/arxiv.2008.11193

Cited by 9 publications

(25 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Correlation between Gradient Norm and Ease of Reconstruction Following on from Appendix C, we investigate the relationship between reconstruction and the gradient norm of loss with respected to released model parameters computed on the target point through training. Recent work on training data memorization [2] and individual privacy accounting in differential privacy [68] have used the gradient norm of a model with respect to the loss induced by a training point as a measure of memorization or privacy leakage. In Figure 21, we evaluate the MSE between target and reconstructions for each 1K target point on MNIST, and also plot the sum of gradient norms over training.…”

Section: K Fine-grained Analysis Of Cifar-10 Reconstructions Over Rel...mentioning

confidence: 99%

Reconstructing Training Data with Informed Adversaries

Balle¹,

Cherubin²,

Hayes³

2022

Preprint

View full text Add to dashboard Cite

Given access to a machine learning model, can an adversary reconstruct the model's training data? This work studies this question from the lens of a powerful informed adversary who knows all the training data points except one. By instantiating concrete attacks, we show it is feasible to reconstruct the remaining data point in this stringent threat model. For convex models (e.g. logistic regression), reconstruction attacks are simple and can be derived in closed-form. For more general models (e.g. neural networks), we propose an attack strategy based on training a reconstructor network that receives as input the weights of the model under attack and produces as output the target data point. We demonstrate the effectiveness of our attack on image classifiers trained on MNIST and CIFAR-10, and systematically investigate which factors of standard machine learning pipelines affect reconstruction success. Finally, we theoretically investigate what amount of differential privacy suffices to mitigate reconstruction attacks by informed adversaries. Our work provides an effective reconstruction attack that model developers can use to assess memorization of individual points in general settings beyond those considered in previous works (e.g. generative language models or access to training gradients); it shows that standard models have the capacity to store enough information to enable high-fidelity reconstruction of training data points; and it demonstrates that differential privacy can successfully mitigate such attacks in a parameter regime where utility degradation is minimal.

show abstract

Section: K Fine-grained Analysis Of Cifar-10 Reconstructions Over Rel...mentioning

confidence: 99%

Reconstructing Training Data with Informed Adversaries

Balle¹,

Cherubin²,

Hayes³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In addition, Bayesian differential privacy (Triastcyn & Faltings, 2020) provides data-dependent privacy guarantees that afford strong protection to "typical" data by making distributional assumptions about the sensitive data. The Rényi-DP-based privacy filters of (Feldman & Zrnic, 2020) are also closely related to our work; the authors study composition of personalized (but not per-instance) privacy losses using adaptively-chosen privacy parameters.…”

Section: Related Workmentioning

confidence: 99%

Privately Publishable Per-instance Privacy

Redberg¹,

Wang²

2021

Preprint

View full text Add to dashboard Cite

We consider how to privately share the personalized privacy losses incurred by objective perturbation, using per-instance differential privacy (pDP). Standard differential privacy (DP) gives us a worst-case bound that might be orders of magnitude larger than the privacy loss to a particular individual relative to a fixed dataset. The pDP framework provides a more fine-grained analysis of the privacy guarantee to a target individual, but the per-instance privacy loss itself might be a function of sensitive data. In this paper, we analyze the per-instance privacy loss of releasing a private empirical risk minimizer learned via objective perturbation, and propose a group of methods to privately and accurately publish the pDP losses at little to no additional privacy cost.

show abstract

“…Transparent and trustworthy data processing systems must therefore be capable of not only accounting for individual privacy loss (e.g. shown in work on individual Rényi DP (RDP) [2]), but to also selectively apply privacypreserving mechanisms to specific private attributes [3]. So far, few works have investigated the contribution of individual attributes to the query function's sensitivity, and therefore to overall privacy loss.…”

Section: Introductionmentioning

confidence: 99%

“…As, however, the realised gradient norm of a query may be considerably lower, the outlook of this type of privacy accounting may be unnecessarily pessimistic. Individual privacy accounting, and specifically the work by Feldman et al [2], proposes an alternative formulation which separately accounts for each individual's influence on the outcome of a computation via the actual gradient norm. The authors express privacy guarantees using Rényi DP [5].…”

Section: Introductionmentioning

confidence: 99%

“…The authors express privacy guarantees using Rényi DP [5]. We note that the utilisation of the gradient norm for individual privacy accounting, which we will be using in this work, is performed in conjunction with a privacy filter or odometer presented in [2]. Heterogeneous DP [3] captures the variation of privacy expectations across information held by one individual and uses a modified Laplace mechanism.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Partial sensitivity analysis in differential privacy

Mueller¹,

Ziller²,

Usynin³

et al. 2021

Preprint

View full text Add to dashboard Cite

Differential privacy (DP) allows the quantification of privacy loss when the data of individuals is subjected to algorithmic processing such as machine learning, as well as the provision of objective privacy guarantees. However, while techniques such as individual Rényi DP (RDP) allow for granular, per-person privacy accounting, few works have investigated the impact of each input feature on the individual's privacy loss. Here we extend the view of individual RDP by introducing a new concept we call partial sensitivity, which leverages symbolic automatic differentiation to determine the influence of each input feature on the gradient norm of a function. We experimentally evaluate our approach on queries over private databases, where we obtain a feature-level contribution of private attributes to the DP guarantee of individuals. Furthermore, we explore our findings in the context of neural network training on synthetic data by investigating the partial sensitivity of input pixels on an image classification task.Preprint. Under review.

show abstract

Individual Privacy Accounting via a Renyi Filter

Cited by 9 publications

References 0 publications

Reconstructing Training Data with Informed Adversaries

Reconstructing Training Data with Informed Adversaries

Privately Publishable Per-instance Privacy

Partial sensitivity analysis in differential privacy

Contact Info

Product

Resources

About