Evaluating the utility of synthetic COVID-19 case data

Abstract

Background Concerns about patient privacy have limited access to COVID-19 datasets. Data synthesis is one approach for making such data broadly available to the research community in a privacy protective manner. Objectives Evaluate the utility of synthetic data by comparing analysis results between real and synthetic data. Methods A gradient boosted classification tree was built to predict death using Ontario’s 90 514 COVID-19 case records linked with community comorbidity, demographic, and socioeconomic characteristics. Model accuracy and relationships were evaluated, as well as privacy risks. The same model was developed on a synthesized dataset and compared to one from the original data. Results The AUROC and AUPRC for the real data model were 0.945 [95% confidence interval (CI), 0.941–0.948] and 0.34 (95% CI, 0.313–0.368), respectively. The synthetic data model had AUROC and AUPRC of 0.94 (95% CI, 0.936–0.944) and 0.313 (95% CI, 0.286–0.342) with confidence interval overlap of 45.05% and 52.02% when compared with the real data. The most important predictors of death for the real and synthetic models were in descending order: age, days since January 1, 2020, type of exposure, and gender. The functional relationships were similar between the two data sets. Attribute disclosure risks were 0.0585, and membership disclosure risk was low. Conclusions This synthetic dataset could be used as a proxy for the real dataset.

Demonstrating an approach for evaluating synthetic geospatial and temporal epidemiologic data utility: Results from analyzing >1.8 million SARS-CoV-2 tests in the United States National COVID Cohort Collaborative (N3C)

“…The data used in this manuscript do not reflect the current size nor state of the N3C LDS. Other statistical techniques such as equivalence testing, bhattacharyya distance [50,51], or adversarial challenges [28] could be used in the future to compare similarity between epidemic curves. The Wilcoxon signed-rank and paired t-tests assume the null hypothesis that the original and synthetic datasets are equivalent.…”

“…Therefore, it is important to evaluate N3C synthetic data in a manner that can inform users with a wide range of intended use cases and definitions for synthetic data fitness for use. [25] The utility of synthetic health data has been evaluated in other work [15,19,20,[26][27][28][29][30] outside of N3C which applied a variety of the ways one can validate synthetic data. [31] However, N3C synthetic data utility has only been evaluated once before.…”

Synthetic data in machine learning for medicine and healthcare

“…The advancements made are maturing so rapidly that we should carefully understand what control we cede if we allow for 'spurious imitations' to gain a foothold in healthcare decision-making. For instance, since the start of the COVID-19 pandemic, there has been an explosion of interest around the development of synthetic data, with use cases such as the training of AI algorithms 17,52 , epidemiological modelling and digital contact tracing [53][54][55] , and data sharing between hospitals 56 . Because synthetic data will undoubtedly soon be used to solve pressing problems in healthcare, it is urgent to develop and refine regulatory frameworks involving synthetic data and the monitoring of their impact in society.…”

The Problem of Fairness in Synthetic Healthcare Data

“…Access to the data can enable exploration of alternative strategies to combat COVID-19 spread [ 8 , 9 ]. Synthetic COVID-19 healthcare datasets can come to our rescue and have been shown to be useful as proxies for real data [ 10 ].…”