Validating Seed Data Samples for Synthetic Identities – Methodology and Uniqueness Metrics

Abstract

This work explores the identity attribute of synthetic face samples derived from Generative Adversarial Networks. The goal is to determine if individual samples are unique in terms of identity, firstly with respect to the seed dataset that trains the GAN model and secondly with respect to other synthetic face samples. Two approaches are introduced to enable the comparative analysis of large sets of synthetic face samples. The first of these uses ROC curves to determine identity uniqueness using a number of large publicly available datasets of real facial samples to provide reference ROCs as a baseline. The second approach uses a thresholding technique utilizing again large publicly available datasets as a reference. For this approach, new metrics are introduced, and a technique is provided to remove the most connected data samples within a large synthetic dataset. The remaining synthetic samples can be considered as unique as data samples gathered from different real individuals. Several StyleGAN models are used to create the synthetic datasets, and variations in key model parameters are explored. It is concluded that the resulting synthetic data samples exhibit excellent uniqueness when compared with the original training dataset, but significantly less uniqueness when comparisons are made within the synthetic dataset. Nevertheless, it is possible to remove the most highly connected synthetic data samples. Thus, in some cases, up to 92% of the data samples in a 20k synthetic dataset can be shown to exhibit similar uniqueness to data samples taken from real public datasets.

On the use of automatically generated synthetic image datasets for benchmarking face recognition

“…This indicates there is a higher density of lookalikes between Sy and Se than in a real population, which is here modeled by IJB-C. This is similar to the observation made in [32] using StyleGAN. In the article, they notice that this is caused by the presence of children in FFHQ, and thus also in StyleGAN's output distribution, while SOTA face recognition networks are typically not trained on faces of children and thus perform poorly on this population.…”

“…To assess the requirement of privacy, we need to verify that generated identities do not simply reproduce existing identities from the FFHQ dataset. We evaluate this by reproducing on our synthetic dataset an experiment originally proposed in [32] on the first version of StyleGAN. It consists in comparing the identity similarity between a synthetic dataset (Sy) and a seed dataset (Se), which is the dataset used to train the face generator.…”

Towards End-to-End Neural Face Authentication in the Wild -- Quantifying and Compensating for Directional Lighting Effects

“…The closer an ROC curve is to unity, the better the performance of the FR model on the selected samples. More information regarding the ROC and its interpretation and use can be found in [40].…”

“…The effect of these augmentations on the performance of a SoA FR method is quantified using Receiver Operating Characteristic curve (ROC) techniques. A similar approach was used recently to validate synthetic facial identities [40]. Note that a re-lighting augmentation approach was adopted as existing public datasets do not provide sufficient lighting variability.…”

Automated Sewer Defects Detection Using Style-Based Generative Adversarial Networks and Fine-Tuned Well-Known CNN Classifier

“…The generator learns to distribute data from a real training dataset and the discriminator is trained to judge whether a sample is a real or a generated sample. The ultimate goal is to train the generator to produce high-quality synthetic images that can fool the discriminator [13,14,19]. At present, the StyleGAN is a representative of the most advanced GAN techniques and can produce sufficiently high-quality and photo-realistic samples with relatively low computation cost [13,20,21].…”