Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures

Fredrikson, Matt; Jha, Somesh; Ristenpart, Thomas

doi:10.1145/2810103.2813677

Cited by 2,037 publications

(1,749 citation statements)

References 20 publications

Supporting

Mentioning

1,609

Contrasting

Unclassified

Order By: Relevance

“…Model inversion has also been applied to face recognition models [16]. In this scenario, the model's output is set to 1 for class i and 0 for the rest, and model inversion is used to construct an input that produces these outputs.…”

Section: Related Workmentioning

confidence: 99%

“…If the images in a class are diverse (e.g., if the class contains multiple individuals or many different objects), the results of model inversion as used in [16] are semantically meaningless and not recognizable as any specific image from the training dataset. To illustrate this, we ran model inversion against a convolutional neural network 13 trained on the CIFAR-10 dataset, which is a standard benchmark for object recognition models.…”

Section: Related Workmentioning

confidence: 99%

“…Inferring information about the model's training dataset should not be confused with techniques such as model inversion that use a model's output on a hidden input to infer something about this input [17] or to extract features that characterize one of the model's classes [16]. As explained in [27] and Section IX, model inversion does not produce an actual member of the model's training dataset, nor, given a record, does it infer whether this record was in the training dataset.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Membership Inference Attacks Against Machine Learning Models

Shokri

Stronati

Song

et al. 2017

2017 IEEE Symposium on Security and Privacy (SP)

2,767

2,817

View full text Add to dashboard Cite

Abstract-We quantitatively investigate how machine learning models leak information about the individual data records on which they were trained. We focus on the basic membership inference attack: given a data record and black-box access to a model, determine if the record was in the model's training dataset. To perform membership inference against a target model, we make adversarial use of machine learning and train our own inference model to recognize differences in the target model's predictions on the inputs that it trained on versus the inputs that it did not train on.We empirically evaluate our inference techniques on classification models trained by commercial "machine learning as a service" providers such as Google and Amazon. Using realistic datasets and classification tasks, including a hospital discharge dataset whose membership is sensitive from the privacy perspective, we show that these models can be vulnerable to membership inference attacks. We then investigate the factors that influence this leakage and evaluate mitigation strategies.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Membership Inference Attacks Against Machine Learning Models

Shokri

Stronati

Song

et al. 2017

2017 IEEE Symposium on Security and Privacy (SP)

2,767

2,817

View full text Add to dashboard Cite

show abstract

“…The shared parameters were averaged and passed back to the entities for the next iteration. Model inversion attacks were demonstrated in [11]. The authors showed that shared models leak information and are vulnerable even against a "black-box" adversary (that interacts with the model only via inputs and outputs).…”

Section: B Prior Approaches For Privacy-aware Distributed Learningmentioning

confidence: 99%

Deep learning for situational understanding

Chakraborty

Preece

Alzantot

et al. 2017

2017 20th International Conference on Information Fusion (Fusion)

View full text Add to dashboard Cite

Abstract-Situational understanding (SU) requires a combination of insight -the ability to accurately perceive an existing situation -and foresight -the ability to anticipate how an existing situation may develop in the future. SU involves information fusion as well as model representation and inference. Commonly, heterogenous data sources must be exploited in the fusion process: often including both hard and soft data products. In a coalition context, data and processing resources will also be distributed and subjected to restrictions on information sharing. It will often be necessary for a human to be in the loop in SU processes, to provide key input and guidance, and to interpret outputs in a way that necessitates a degree of transparency in the processing: systems cannot be "black boxes". In this paper, we characterize the Coalition Situational Understanding (CSU) problem in terms of fusion, temporal, distributed, and human requirements. There is currently significant interest in deep learning (DL) approaches for processing both hard and soft data. We analyze the state-of-the-art in DL in relation to these requirements for CSU, and identify areas where there is currently considerable promise, and key gaps.

show abstract

“…Here, the output function is a list of (anonymized) inputs, whereas we consider arithmetic computations leading to relatively small outputs. In model inversion [17], machine-learning algorithms are used to deduce sensitive personal information from various outputs. This is different from finding personal inputs to a specific output function.…”

Section: Related Workmentioning

confidence: 99%

Improved privacy of dynamic group services

Veugen

Doumen

Erkin

et al. 2017

EURASIP J. on Info. Security

View full text Add to dashboard Cite

We consider dynamic group services, where outputs based on small samples of privacy-sensitive user inputs are repetitively computed. The leakage of user input data is analysed, caused by producing multiple outputs, resulting from inputs of frequently changing sets of users. A cryptographic technique, known as random user selection, is investigated. We show the effect of random user selection, given different types of output functions, thereby disproving earlier work. A new security measure is introduced, which provably improves the privacy-preserving effect of random user selection, irrespective of the output function. We show how this new security measure can be implemented in existing cryptographic protocols. To investigate the effectiveness of our security measure, we conducted a couple of statistical simulations with large user populations, which show that it forms a key ingredient, at least for the output function addition. Without it, an adversary is able to determine a user input, with increasing accuracy when more outputs become available. When the security measure is implemented, an adversary remains oblivious of user inputs, even when thousands of outputs are collected. Therefore, our new security measure assures that random user selection is an effective way of protecting the privacy of dynamic group services.

show abstract

Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures

Cited by 2,037 publications

References 20 publications

Membership Inference Attacks Against Machine Learning Models

Membership Inference Attacks Against Machine Learning Models

Deep learning for situational understanding

Improved privacy of dynamic group services

Contact Info

Product

Resources

About