Generation and evaluation of synthetic patient data

Abstract

Background: Machine learning (ML) has made a significant impact in medicine and cancer research; however, its impact in these areas has been undeniably slower and more limited than in other application domains. A major reason for this has been the lack of availability of patient data to the broader ML research community, in large part due to patient privacy protection concerns. High-quality, realistic, synthetic datasets can be leveraged to accelerate methodological developments in medicine. By and large, medical data is high dimensional and often categorical. These characteristics pose multiple modeling challenges. Methods: In this paper, we evaluate three classes of synthetic data generation approaches; probabilistic models, classification-based imputation models, and generative adversarial neural networks. Metrics for evaluating the quality of the generated synthetic datasets are presented and discussed. Results: While the results and discussions are broadly applicable to medical data, for demonstration purposes we generate synthetic datasets for cancer based on the publicly available cancer registry data from the Surveillance Epidemiology and End Results (SEER) program. Specifically, our cohort consists of breast, respiratory, and non-solid cancer cases diagnosed between 2010 and 2015, which includes over 360,000 individual cases. Conclusions: We discuss the trade-offs of the different methods and metrics, providing guidance on considerations for the generation and usage of medical synthetic data.

Synthetic Observational Health Data with GANs: from slow adoption to a boom in medical research and ultimately digital twins?

“…Non-symmetric measure of difference between two PDs, related to relative entropy. Given a feature X, p(x) and q(x) the PD of the real and synthetic data respectively, the KLD of q(x) from p(x) is the amount of information lost when q(x) is trained to estimate p(x) (Jiawei, 2018;Goncalves et al, 2020).…”

“…Two predictors are trained to predict the label, one from the synthetic data and another from a portion of the real data. Their performance is compared on the left out real data (Choi et al, 2017a;Camino et al, 2018;Goncalves et al, 2020;Tantipongpipat et al, 2019;Baowaly et al, 2019).…”

“…Some authors conducted a privacy risk assessment to evaluate the risk of reidentification. The empirical analyses were based on the definitions of MI, AD (Choi et al, 2017a;Goncalves et al, 2020;Chen et al, 2019b;Chin-Cheong et al, 2020) and the Reproduction rate (RR) . Cosine similarities between pairs of samples was also used (Torfi and Beyki, 2019).…”

“…Most studies report low success rates for these types of attacks, and little effect from the sample size, although Chen et al note that sample sizes under 10k lead to higher risk. Goncalves et al evaluated MC-medGAN against multiple non-adversarial generative models in a variety of privacy compromising attacks, including AD, obtaining inconsistent results for MC-medGAN (Goncalves et al, 2020). While this is not mentioned by the authors, multiple results reported in the publication point to the fact that the GAN was not properly trained or suffered mode-collapse.In black-box and white-box type attacks, including the LOGAN (Hayes et al, 2017) method, medGAN performed considerably better than WGAN-GP (Chen et al, 2019b), the algorithm which served as basis for improvements to medGAN in publications discussed in Section 3.4.1.…”

Synthetic Observational Health Data with GANs: from slow adoption to a boom in medical research and ultimately digital twins?

“…Inter-dimensional correlation Dimension-wise Pearson coefficient correlation matrices for both real and synthetic data (Beaulieu-Jones et al, 2019;Goncalves et al, 2020;Torfi and Beyki, 2019;Frid-Adar et al, 2018;Ozyigit et al, 2020;Yang et al, 2019c;Zhu et al, 2020a;Walsh et al, 2020;Yale et al, 2019a;Ozyigit et al, 2020;Dash et al, 2019;Bae et al, 2020b).…”