Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' Outputs

Abstract

It is known that deep neural networks, trained for the classification of a non-sensitive target attribute, can reveal sensitive attributes of their input data; through features of different granularity extracted by the classifier. We, taking a step forward, show that deep classifiers can be trained to secretly encode a sensitive attribute of users' input data, at inference time, into the classifier's outputs for the target attribute. An attack that works even if users have a white-box view of the classifier, and can keep all internal representations hidden except for the classifier's estimation of the target attribute. We introduce an information-theoretical formulation of such adversaries and present efficient empirical implementations for training honest-but-curious (HBC) classifiers based on this formulation: deep models that can be accurate in predicting the target attribute, but also can utilize their outputs to secretly encode a sensitive attribute. Our evaluations on several tasks in real-world datasets show that a semi-trusted server can build a classifier that is not only perfectly honest but also accurately curious. Our work highlights a vulnerability that can be exploited by malicious machine learning service providers to attack their user's privacy in several seemingly safe scenarios; such as encrypted inferences, computations at the edge, or private knowledge distillation. We conclude by showing the difficulties in distinguishing between standard and HBC classifiers and discussing potential proactive defenses against this vulnerability of deep classifiers.

Finding MNEMON: Reviving Memories of Node Embeddings

“…The attackers can then obtain the node embedding matrix from the data holder through the rouge provider. This attack scenario is in line with the malicious machine learning provider scenario discussed by Song et al [65] and Malekzadeh et al [47].…”

“…They cannot interact with the node embedding models since such pipelines usually operate in one direction. For instance, the data holder may have integrated with the malicious machine learning solution providers (i.e., MLaaS providers) from the AWS Marketplace [47,65], or the data holder is part of a vertical federated learning environment in an enterprise [71]. In both cases, the node embeddings are part of the learning process and can be obtained by either the malicious MLaaS providers [47,65] or the insiders [71] in the pipeline.…”

Dikaios: Privacy Auditing of Algorithmic Fairness via Attribute Inference Attacks

“…Attribute inference attack is closer to the problem of data amputation where given non-sensitive attributes, the goal is the predict the sensitive attribute [11,14,15,41]. Malekzadeh et al [29] leverage an informational theoretic view to infer the sensitive attributes from output predictions. However, their setting is different where they consider a malicious model designer who injects sensitive attribute to be inferred later after deployment.…”

You Don't Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers' Private Personas

“…Furthermore, given black-box access to a language model's pre-train and finetune stages, Zanella-Béguelin et al (2020) showed that sensitive sequences of the fine-tuning dataset can be extracted. For the distributed client-server setup, Malekzadeh et al (2021) considered the sensitive attribute leakage from the server side with honest-but-curious (HBC) classifiers.…”