An empirical study on evaluation metrics of generative adversarial networks

Xu, Qiantong; Huang, Gao; Yang, Yuan; Guo, Chuan; Sun, Yu; Wu, Felix; Weinberger, Kilian Q.

doi:10.48550/arxiv.1806.07755

Cited by 71 publications

(69 citation statements)

References 22 publications

(36 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our work is closely related to other works benchmarking evaluation metrics of generative models (O'Bray et al, 2021;Xu et al, 2018). A consistent method for assessing evaluation metrics is to start with the creation of "reference" and "generated" sets S r and S g that originate from the same real distribution P r .…”

Section: Benchmarking Evaluation Metricsmentioning

confidence: 92%

“…The linear kernel k(x i , x j ) = x i • x j is another parameter-free kernel used with MMD to evaluate generative models (O'Bray et al, 2021). In addition, the RBF kernel (Equation 2) with d(•, •) as the Euclidean distance is widely used (Xu et al, 2018;Gretton et al, 2006). The choice of σ in Equation 2 has a significant impact on the output of the RBF kernel, and methods for finding and selecting an optimal value is an important area of research (Bach et al, 2004;Gretton et al, 2012a;b).…”

Section: Neural-network-based Metricsmentioning

confidence: 99%

“…Then, one gradually increases distortion to the generated set while recording each metric's response to the distortion. In the image domain, Xu et al (2018) Although objective, evaluation using Pearson correlation is biased in that the relationship between the metric and degree of perturbation need not be linear.…”

Section: Benchmarking Evaluation Metricsmentioning

confidence: 99%

“…In image generation evaluation, classifiers trained on ImageNet (Deng et al, 2009) are frequently used to extract image embeddings (Bińkowski et al, 2018;Heusel et al, 2017;Kynkäänniemi et al, 2019;Xu et al, 2018;Naeem et al, 2020). While classifiers such as Inception v3 (Szegedy et al, 2016) are consistently used, recent works have investigated the use of randomly-initialized CNNs with no further training (hereafter referred to as a random network) in generative model evaluation.…”

Section: Introductionmentioning

confidence: 99%

“…While classifiers such as Inception v3 (Szegedy et al, 2016) are consistently used, recent works have investigated the use of randomly-initialized CNNs with no further training (hereafter referred to as a random network) in generative model evaluation. Xu et al (2018); Naeem et al (2020) found that a random CNN performs similarly to ImageNet classifiers on natural images and is superior outside of the natural image domain. In the graph domain, random GNNs have been shown to extract meaningful features to solve downstream graph tasks without training (Kipf & Welling, 2017;Morris et al, 2019;Xu et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

On Evaluation Metrics for Graph Generative Models

Thompson¹,

Knyazev²,

Ghalebi³

et al. 2022

Preprint

View full text Add to dashboard Cite

In image generation, generative models can be evaluated naturally by visually inspecting model outputs. However, this is not always the case for graph generative models (GGMs), making their evaluation challenging. Currently, the standard process for evaluating GGMs suffers from three critical limitations: i) it does not produce a single score which makes model selection challenging, ii) in many cases it fails to consider underlying edge and node features, and iii) it is prohibitively slow to perform. In this work, we mitigate these issues by searching for scalar, domain-agnostic, and scalable metrics for evaluating and ranking GGMs.To this end, we study existing GGM metrics and neural-network-based metrics emerging from generative models of images that use embeddings extracted from a task-specific network. Motivated by the power of certain Graph Neural Networks (GNNs) to extract meaningful graph representations without any training, we introduce several metrics based on the features extracted by an untrained random GNN. We design experiments to thoroughly test metrics on their ability to measure the diversity and fidelity of generated graphs, as well as their sample and computational efficiency. Depending on the quantity of samples, we recommend one of two random-GNN-based metrics that we show to be more expressive than pre-existing metrics. While we focus on applying these metrics to GGM evaluation, in practice this enables the ability to easily compute the dissimilarity between any two sets of graphs regardless of domain. Our code is released at: https://github.com/uoguelph-mlrg/GGM-metrics.

show abstract

Section: Benchmarking Evaluation Metricsmentioning

confidence: 92%

Section: Neural-network-based Metricsmentioning

confidence: 99%

Section: Benchmarking Evaluation Metricsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

On Evaluation Metrics for Graph Generative Models

Thompson¹,

Knyazev²,

Ghalebi³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Recent Developments in Generative Adversarial Networks

Singh,

Parashar

2023

Artificial Intelligence and Sustainable Computing

View full text Add to dashboard Cite

Medical image data augmentation: techniques, comparisons and interpretations

Göçeri

2023

Artif Intell Rev

View full text Add to dashboard Cite

Designing deep learning based methods with medical images has always been an attractive area of research to assist clinicians in rapid examination and accurate diagnosis. Those methods need a large number of datasets including all variations in their training stages. On the other hand, medical images are always scarce due to several reasons, such as not enough patients for some diseases, patients do not want to allow their images to be used, lack of medical equipment or equipment, inability to obtain images that meet the desired criteria. This issue leads to bias in datasets, overfitting, and inaccurate results. Data augmentation is a common solution to overcome this issue and various augmentation techniques have been applied to different types of images in the literature. However, it is not clear which data augmentation technique provides more efficient results for which image type since different diseases are handled, different network architectures are used, and these architectures are trained and tested with different numbers of data sets in the literature. Therefore, in this work, the augmentation techniques used to improve performances of deep learning based diagnosis of the diseases in different organs (brain, lung, breast, and eye) from different imaging modalities (MR, CT, mammography, and fundoscopy) have been examined. Also, the most commonly used augmentation methods have been implemented, and their effectiveness in classifications with a deep network has been discussed based on quantitative performance evaluations. Experiments indicated that augmentation techniques should be chosen carefully according to image types.

show abstract

An empirical study on evaluation metrics of generative adversarial networks

Cited by 71 publications

References 22 publications

On Evaluation Metrics for Graph Generative Models

On Evaluation Metrics for Graph Generative Models

Recent Developments in Generative Adversarial Networks

Medical image data augmentation: techniques, comparisons and interpretations

Contact Info

Product

Resources

About