Generation and Evaluation of Synthetic Data in a University Hospital Setting

Kaabachi, Bayrem; Despraz, Jérémie; Meurers, Thierry; Praßer, Fabian; Raisaro, Jean Louis

doi:10.3233/shti220420

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From the papers that did include a privacy evaluation, 84% (20/24) mainly relied on dataset-based evaluation. A smaller number, 8% (2/24), focused on based on the model or mechanism itself, such as those that exploit GAN architectures 30 or those that involve a shadow modeling process [31][32][33] and another 8& (2/24) performed both evaluations.…”

Section: Synthetic Data Privacymentioning

confidence: 99%

Can We Trust Synthetic Data in Medicine? A Scoping Review of Privacy and Utility Metrics

Kaabachi,

Despraz,

Meurers

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

IntroductionSharing and re-using health-related data beyond the scope of its initial collection is essential for accelerating research, developing robust and trustworthy machine learning algorithms methods that can be translated into clinical settings. The sharing of synthetic data, artificially generated to resemble real patient data, is increasingly recognized as a promising means to enable such a re-use while addressing the privacy concerns related to personal medical data. Nonetheless, no consensus exists yet on a standard approach for systematically and quantitatively evaluating the actual privacy gain and residual utility of synthetic data, de-facto hindering its adoption.ObjectiveIn this work, we present and systematize current knowledge on the field of synthetic health-related data evaluation both in terms of privacy and utility. We provide insights and critical analysis into the current state of the art and propose concrete directions and steps forward for the research community.MethodsWe assess and contextualize existing knowledge in the field through a scoping review and the creation of a common ontology that encompasses all the methods and metrics used to assess synthetic data. We follow the PRISMA-ScR methodology in order to perform data collection and knowledge synthesis.ResultsWe include 92 studies in the scoping review. We analyze and classify them according to the proposed ontology. We found 48 different methods to evaluate the residual statistical utility of synthetic data and 9 methods that are used to evaluate the residual privacy risks. Moreover, we observe that there is currently no consensus among researchers regarding neither individual metrics nor family of metrics for evaluating the privacy and utility of synthetic data. Our findings on the privacy of synthetic data show that there is an alarming tendency to trust the safety of synthetic data without properly evaluating it.ConclusionAlthough the use of synthetic data in healthcare promises to offer an easy and hassle-free alternative to real data, the lack of consensus in terms of evaluation hinders the adoption of this new technology. We believe that, by raising awareness and providing a comprehensive taxonomy on evaluation methods that takes into account the current state of literature, our work can foster the development and adoption of uniform approaches and consequently facilitate the use of synthetic data in the medical domain.

show abstract