A comparison of synthetic data generation and federated analysis for enabling international evaluations of cardiovascular health

Abstract

Sharing health data for research purposes across international jurisdictions has been a challenge due to privacy concerns. Two privacy enhancing technologies that can enable such sharing are synthetic data generation (SDG) and federated analysis, but their relative strengths and weaknesses have not been evaluated thus far. In this study we compared SDG with federated analysis to enable such international comparative studies. The objective of the analysis was to assess country-level differences in the role of sex on cardiovascular health (CVH) using a pooled dataset of Canadian and Austrian individuals. The Canadian data was synthesized and sent to the Austrian team for analysis. The utility of the pooled (synthetic Canadian + real Austrian) dataset was evaluated by comparing the regression results from the two approaches. The privacy of the Canadian synthetic data was assessed using a membership disclosure test which showed an F1 score of 0.001, indicating low privacy risk. The outcome variable of interest was CVH, calculated through a modified CANHEART index. The main and interaction effect parameter estimates of the federated and pooled analyses were consistent and directionally the same. It took approximately one month to set up the synthetic data generation platform and generate the synthetic data, whereas it took over 1.5 years to set up the federated analysis system. Synthetic data generation can be an efficient and effective tool for enabling multi-jurisdictional studies while addressing privacy concerns.

An evaluation of the replicability of analyses using synthetic health data

“…For the N0147 dataset, we evaluated the impact of bowel obstruction on 5 year survival as a binary outcome 65 . The CCHS model we constructed evaluated cardiovascular health using the CANHEART index 66 , which was dichotomized at the “poor” to “intermediate” health boundary, and the covariate of interest was sex 67 . The DCCG model we constructed examines the relationship between sex and medical complications 68 , 69 .…”

“…We would want this to be as small as possible Privacy The membership disclosure metric computed on the pooled datasets for that value of m 95 . The acceptable threshold for this relative F1 score metric is 0.2 67 , 94 , 95 …”

“…The metric is a relative F1 score that evaluates the accuracy of such adversary attacks compared to a naïve attack which does not use the information in the synthetic data. Previous work has used a threshold of 0.2 to determine if the relative F1 score was low enough 67 , 94 , 95 .…”