Deep Generative Models for Synthetic Data: A Survey

Abstract

A growing interest in synthetic data has stimulated the development and advancement of a large variety of deep generative models for a wide range of applications. However, as this research has progressed, its streams have become more specialized and disconnected from one another. This is why models for synthesizing text data for natural language processing cannot readily be compared to models for synthesizing health records anymore. To mitigate this isolation, we propose a data-driven evaluation framework for generative models for synthetic sequential data, an important and challenging sub-category of synthetic data, based on five high-level criteria: representativeness, novelty, realism, diversity and coherence of a synthetic data-set relative to the original data-set regardless of the models' internal structures. The criteria reflect requirements different domains impose on synthetic data and allow model users to assess the quality of synthetic data across models. In a critical review of generative models for sequential data, we examine and compare the importance of each performance criterion in numerous domains. We find that realism and coherence are more important for synthetic data natural language, speech and audio processing tasks. At the same time, novelty and representativeness are more important for healthcare and mobility data. We also find that measurement of representativeness is often accomplished using statistical metrics, realism by using human judgement, and novelty using privacy tests.

Evaluation is key: a survey on evaluation measures for synthetic time series

“…Lately, Eigenschink et al [15] proposed an evaluation framework for (mainly deep) generative models for synthesizing all kinds of sequential data: Text, audio, video, and time series. They aimed to overcome the isolated evaluation of generators in these areas and present a universally applicable set of criteria to check.…”

“…Most address specific domains or data types other than time series [12,14,18,19], while a transfer of findings is non-trivial. Another one addresses the evaluation of sequential data generation, but in too broad a scope to provide detailed insights for time series [15]. Others are tailored towards images or are data type-agnostic but limit their scope to GANs [4,21].…”

“…Among all pairs of orange and cyan points, x and y are the closest in cosine similarity Fig. 15 Procedure and artifacts for a MIA. This diagram sketches the procedure and artifacts for a MIA.…”

Evaluation is Key: A Survey on Evaluation Measures for Synthetic Time Series

“…Most of them address specific domains or data types other than time series (Iqbal and Qureshi (2022); Fatima et al (2022); Ji et al (2020); Xu et al (2018)), while a transfer of findings is non-trivial. Another one addresses the evaluation of sequential data generation, but in too broad a scope and superficial to provide detailed insights for this time series (Eigenschink et al (2023)). Others are tailored towards images or are data typeagnostic but limit their scope to GANs (Brophy et al (2023); Theis et al (2016)).…”

Evaluating the Impact of Health Care Data Completeness for Deep Generative Models

“…DGMs are a family of deep learning architectures that use deep neural networks to infer high dimensional distributions from large datasets. 6 They have displayed state-of-the-art performance across a variety of tasks where mapping these complex distributions is essential, such as speech generation and compression. 7 Similarly, DGMs have increasingly been applied in health care to generate synthetic data sampled from the inferred probability distribution as a means of augmenting existing datasets or creating synthetic, fully anonymous datasets.…”