Membership Inference Attacks From First Principles

Abstract

We introduce the first model-stealing attack that extracts precise, nontrivial information from black-box production language models like Ope-nAI's ChatGPT or Google's PaLM-2. Specifically, our attack recovers the embedding projection layer (up to symmetries) of a transformer model, given typical API access. For under $20 USD, our attack extracts the entire projection matrix of OpenAI's ada and babbage language models. We thereby confirm, for the first time, that these black-box models have a hidden dimension of 1024 and 2048, respectively. We also recover the exact hidden dimension size of the gpt-3.5-turbo model, and estimate it would cost under $2,000 in queries to recover the entire projection matrix. We conclude with potential defenses and mitigations, and discuss the implications of possible future work that could extend our attack.

Privacy for Free: How does Dataset Condensation Help Privacy?

“…The conclusions are further analytically and empirically generalized to non-linear feature extractors. Then, we empirically validate that models trained on DC-synthesized data are robust to both vanilla loss-based MIA and the state-of-the-art likelihood-based MIA (Carlini et al, 2022). Finally, we study the visual privacy of DC-synthesized data in case of adversary's direct matching attack.…”

“…For LiRA (Carlini et al, 2022), we repeat the preparation of synthetic dataset N m times with different random seeds, and obtain N m shadow T , S and f S . We set N m = 256 for DM and N m = 64 for KIP because of its lower training efficiency.…”

“…Likelihood-based MIA. Recent works (Carlini et al, 2022;Rezaei & Liu, 2021) 2 that are not trained on it (called OUT models). The adversary then measures the means µ in , µ out and the variances σ 2 in , σ 2 out of model confidence for IN and OUT models, respectively.…”

“…However, as we demonstrate in our experiments (Section 5.2), the membership of data used for DC initialization can still be inferred by vanilla loss-based MIA. One countermeasure is to choose hard-to-infer samples (Carlini et al, 2022), i.e., samples whose model outputs are not affected by the membership, as initialization data.…”

“…LiRA is a more powerful MIA because it can achieve higher TPR at low FPR (Carlini et al, 2022), while the adversary's computational cost is higher. Figure 2 provides the ROC curves of LiRA against f S .…”