Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting

Yeom, Samuel; Giacomelli, Irene; Fredrikson, Matt; Jha, Somesh

doi:10.1109/csf.2018.00027

Cited by 633 publications

(893 citation statements)

References 34 publications

Supporting

Mentioning

787

Contrasting

Order By: Relevance

“…We demonstrate that all methods indeed increase the model's membership inference risk. By defining the membership inference advantage as the increase in inference accuracy over random guessing (multiplied by 2) [59], we show that robust machine learning models can incur a membership inference advantage 4.5×, 2.1×, 3.5× times the membership inference advantage of naturally undefended models, on Yale Face, Fashion-MNIST, and CIFAR10 datasets, respectively. (3) We further explore the factors that influence the membership inference performance of the adversarially robust model, including its robustness generalization, the adversarial perturbation constraint, and the model capacity.…”

Section: Introductionmentioning

confidence: 99%

“…For example, participation in a hospital's health analytic training set means that an individual was once a patient in that hospital. It has been shown that the success of membership inference attacks is highly related to the target model's overfitting and sensitivity as to training data [38,42,59]. Adversarially robust models aim to enhance the robustness of target models by ensuring that model predictions are unchanged for a small area (such as l ∞ ball) around each training example.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Privacy Risks of Securing Machine Learning Models against Adversarial Examples

Song

Shokri

Mittal

2019

Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security

179

139

View full text Add to dashboard Cite

The arms race between attacks and defenses for machine learning models has come to a forefront in recent years, in both the security community and the privacy community. However, one big limitation of previous research is that the security domain and the privacy domain have typically been considered separately. It is thus unclear whether the defense methods in one domain will have any unexpected impact on the other domain.In this paper, we take a step towards resolving this limitation by combining the two domains. In particular, we measure the success of membership inference attacks against six state-of-the-art adversarial defense methods that mitigate adversarial examples (i.e., evasion attacks). Membership inference attacks aim to infer an individual's participation in the target model's training set and are known to be correlated with target model's overfitting and sensitivity with regard to training data. Meanwhile, adversarial defense methods aim to enhance the robustness of target models by ensuring that model predictions are unchanged for a small area around each training sample. Thus, adversarial defenses typically have a more fine-grained reliance on the training set and make the target model more vulnerable to membership inference attacks.To perform the membership inference attacks, we leverage the conventional inference method based on prediction confidence and propose two new inference methods that exploit structural properties of adversarially robust defenses. Our experimental evaluation demonstrates that compared with the natural training (undefended) approach, adversarial defense methods can indeed increase the target model's risk against membership inference attacks. When applying adversarial defenses to train the robust models, the membership inference advantage increases by up to 4.5 times compared to the naturally undefended models. Beyond revealing the privacy risks of adversarial defenses, we further investigate the factors that influence the membership information leakage and study the effect of potential countermeasures. 1

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Privacy Risks of Securing Machine Learning Models against Adversarial Examples

Song

Shokri

Mittal

2019

Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security

179

139

View full text Add to dashboard Cite

show abstract

“…The MPLens system is additionally customizable with respect to the attack method, including the shadow model based attack techniques [17], and the threshold-based attack techniques [16]. Furthermore, when using a threshold-based attack, our MPLens system can either accept pre-determined values representing attacker knowledge of the target model error or it can also determine good threshold values through the shadow model training.…”

Section: B Attacker Knowledgementioning

confidence: 99%

Effects of Differential Privacy and Data Skewness on Membership Inference Vulnerability

Truex

Liu

Gürsoy

et al. 2019

2019 First IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA)

View full text Add to dashboard Cite

Membership inference attacks seek to infer the membership of individual training instances of a privately trained model. This paper presents a membership privacy analysis and evaluation system, called MPLens, with three unique contributions. First, through MPLens, we demonstrate how membership inference attack methods can be leveraged in adversarial machine learning. Second, through MPLens, we highlight how the vulnerability of pre-trained models under membership inference attack is not uniform across all classes, particularly when the training data itself is skewed. We show that risk from membership inference attacks is routinely increased when models use skewed training data. Finally, we investigate the effectiveness of differential privacy as a mitigation technique against membership inference attacks. We discuss the trade-offs of implementing such a mitigation strategy with respect to the model complexity, the learning task complexity, the dataset complexity and the privacy parameter settings. Our empirical results reveal that (1) minority groups within skewed datasets display increased risk for membership inference and (2) differential privacy presents many challenging trade-offs as a mitigation technique to membership inference risk.

show abstract

“…Side-channel attacks [22,23] Power consumption, Cryptographic keys processing time, access pattern Membership inference attacks [24][25][26] Confidence scores, gradients Member/Non-member Location inference attacks [27,28] Sensor data on smartphone Location Feature inference attacks [29,30] Partial features, model prediction Missing features CAPTCHA breaking attacks [31][32][33] CAPTCHA Text, audio, etc.…”

Section: Introductionmentioning

confidence: 99%

“…These attacks are also called attribute inference attacks[30]. To distinguish with attribute inference attacks in online social networks, we call them feature inference attacks.…”

mentioning

confidence: 99%

Defending Against Machine Learning Based Inference Attacks via Adversarial Examples: Opportunities and Challenges

Jia

Gong

2020

Adaptive Autonomous Secure Cyber Systems

View full text Add to dashboard Cite

As machine learning (ML) becomes more and more powerful and easily accessible, attackers increasingly leverage ML to perform automated large-scale inference attacks in various domains. In such an ML-equipped inference attack, an attacker has access to some data (called public data) of an individual, a software, or a system; and the attacker uses an ML classifier to automatically infer their private data. Inference attacks pose severe privacy and security threats to individuals and systems. Inference attacks are successful because private data are statistically correlated with public data, and ML classifiers can capture such statistical correlations. In this chapter, we discuss the opportunities and challenges of defending against ML-equipped inference attacks via adversarial examples. Our key observation is that attackers rely on ML classifiers in inference attacks. The adversarial machine learning community has demonstrated that ML classifiers have various vulnerabilities. Therefore, we can turn the vulnerabilities of ML into defenses against inference attacks. For example, ML classifiers are vulnerable to adversarial examples, which add carefully crafted noise to normal examples such that an ML classifier makes predictions for the examples as we desire. To defend against inference attacks, we can add carefully crafted noise into the public data to turn them into adversarial examples, such that attackers' classifiers make incorrect predictions for the private data. However, existing methods to construct adversarial examples are insufficient because they did not consider the unique challenges and requirements for the crafted noise at defending against inference attacks. In this chapter, we take defending against inference attacks in online social networks as an example to illustrate the opportunities and challenges.

show abstract

Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting

Cited by 633 publications

References 34 publications

Privacy Risks of Securing Machine Learning Models against Adversarial Examples

Privacy Risks of Securing Machine Learning Models against Adversarial Examples

Effects of Differential Privacy and Data Skewness on Membership Inference Vulnerability

Defending Against Machine Learning Based Inference Attacks via Adversarial Examples: Opportunities and Challenges

Contact Info

Product

Resources

About