Validating a membership disclosure metric for synthetic health data

Abstract

Background One of the increasingly accepted methods to evaluate the privacy of synthetic data is by measuring the risk of membership disclosure. This is a measure of the F1 accuracy that an adversary would correctly ascertain that a target individual from the same population as the real data is in the dataset used to train the generative model, and is commonly estimated using a data partitioning methodology with a 0.5 partitioning parameter. Objective Validate the membership disclosure F1 score, evaluate and improve the parametrization of the partitioning method, and provide a benchmark for its interpretation. Materials and methods We performed a simulated membership disclosure attack on 4 population datasets: an Ontario COVID-19 dataset, a state hospital discharge dataset, a national health survey, and an international COVID-19 behavioral survey. Two generative methods were evaluated: sequential synthesis and a generative adversarial network. A theoretical analysis and a simulation were used to determine the correct partitioning parameter that would give the same F1 score as a ground truth simulated membership disclosure attack. Results The default 0.5 parameter can give quite inaccurate membership disclosure values. The proportion of records from the training dataset in the attack dataset must be equal to the sampling fraction of the real dataset from the population. The approach is demonstrated on 7 clinical trial datasets. Conclusions Our proposed parameterization, as well as interpretation and generative model training guidance provide a theoretically and empirically grounded basis for evaluating and managing membership disclosure risk for synthetic data.

An evaluation of the replicability of analyses using synthetic health data

“…It is a measure of the precision of the parameter estimate across runs. We would want this to be as small as possible Privacy The membership disclosure metric computed on the pooled datasets for that value of m 95 . The acceptable threshold for this relative F1 score metric is 0.2 67 , 94 , 95 …”

“…Privacy risks were computed using a membership disclosure metric 95 . Membership disclosure evaluates the ability of an adversary to correctly determine if a target individual is in the original data that was used to train the generative model.…”

“…The metric is a relative F1 score that evaluates the accuracy of such adversary attacks compared to a naïve attack which does not use the information in the synthetic data. Previous work has used a threshold of 0.2 to determine if the relative F1 score was low enough 67 , 94 , 95 .…”

“…Therefore, there was a rational process to the choice of metrics. For the third criterion, while we are neutral with respect to the two methods included in our study in that we have evaluated them both before 25 , 94 , 95 , we have also performed more research and applied work with the sequential synthesis method 9 , 28 .…”

Utilization of anonymization techniques to create an external control arm for clinical trial data

“…The data authority that regulates the use of RWD in Finland (Findata) requires the use of k -anonymity ( k = 5), and this was considered as the starting point for the selection of the anonymization framework. Given this premise, two candidate approaches were evaluated using membership inference attacks [ 33 – 35 ]. First, the ε-safe k-anonymization [ 34 ] that may offer slight improvement against membership inference attacks, and it also considers the differential privacy composability problem in the case of multiple data publications.…”