General Facial Representation Learning in a Visual-Linguistic Manner

Zheng, Yinglin; Yang, Hao; Zhang, Ting; Bao, Jianmin; Chen, Dongdong; Huang, Yangyu; Liu, Yuan; Chen, Dong; Zhang, Ming; Wen, Fang

doi:10.1109/cvpr52688.2022.01814

Cited by 62 publications

(52 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to obtain meaningful initial values for the latent codes that will be optimized to create the anonymized version of the real dataset, namely X A , we first pair the real images from the original set (i.e., X R ) with fake ones from the generated dataset (i.e., X F ) in the feature space of the ViT-based FaRL [49] image encoder and use their latent codes for initializing the aforementioned trainable codes. The latent codes of the anonymized dataset are then optimized under the following objectives via two novel loss functions: (a) to be similar to the corresponding real ones, up to a certain margin, using the proposed identity loss (L id ), and (b) to preserve the facial attributes of the corresponding real ones by being pulled closer in the feature space of the pre-trained FaRL [49] image encoder using the proposed attribute preservation loss (L att ). In this way, in contrast to state-of-the-art works [17,28], the anonymized images are optimized to inherit the labels of the original ones.…”

Section: Proposed Methodsmentioning

confidence: 99%

“…To achieve this, in contrast to existing work [17,22,28,45,46] that train custom neural networks from scratch, we propose to work directly in the latent space of a powerful pre-trained GAN, optimizing the latent codes directly with losses that explicitly aim to retain the attributes and obfuscate the identities. More concretely, we use a deep feature-matching loss [49] to match the high-level semantic features between the original and the fake image generated by the latent code, and a margin-based identity loss to control the similarity between the original and the fake image in the ArcFace [9] space. The initialisation of the latent codes is obtained by randomly sampling the latent space of GAN, using them to generate the corresponding synthetic images and finding the nearest neighbors in a semantic space (FARL [49]).…”

Section: Ciagan Deepprivacy Oursmentioning

confidence: 99%

“…More concretely, we use a deep feature-matching loss [49] to match the high-level semantic features between the original and the fake image generated by the latent code, and a margin-based identity loss to control the similarity between the original and the fake image in the ArcFace [9] space. The initialisation of the latent codes is obtained by randomly sampling the latent space of GAN, using them to generate the corresponding synthetic images and finding the nearest neighbors in a semantic space (FARL [49]). In order to preserve texture and pose information of the original image, we perform inversion of the original image and retain the parts that correspond to the properties we want to preserve in the final code.…”

Section: Ciagan Deepprivacy Oursmentioning

confidence: 99%

See 2 more Smart Citations

Attribute-preserving Face Dataset Anonymization via Latent Code Optimization

Barattin¹,

Tzelepis²,

Patras³

et al. 2023

Preprint

View full text Add to dashboard Cite

This work addresses the problem of anonymizing the identity of faces in a dataset of images, such that the privacy of those depicted is not violated, while at the same time the dataset is useful for downstream task such as for training machine learning models. To the best of our knowledge, we are the first to explicitly address this issue and deal with two major drawbacks of the existing state-of-the-art approaches, namely that they (i) require the costly training of additional, purpose-trained neural networks, and/or (ii) fail to retain the facial attributes of the original images in the anonymized counterparts, the preservation of which is of paramount importance for their use in downstream tasks. We accordingly present a task-agnostic anonymization procedure that directly optimizes the images' latent representation in the latent space of a pre-trained GAN. By optimizing the latent codes directly, we ensure both that the identity is of a desired distance away from the original (with an identity obfuscation loss), whilst preserving the facial attributes (using a novel feature-matching loss in FaRL's [49] deep feature space). We demonstrate through a series of both qualitative and quantitative experiments that our method is capable of anonymizing the identity of the images whilstcrucially-better-preserving the facial attributes. We make the code and the pre-trained models publicly available at: https://github.com/chi0tzp/FALCO.

show abstract

Section: Proposed Methodsmentioning

confidence: 99%

Section: Ciagan Deepprivacy Oursmentioning

confidence: 99%

Section: Ciagan Deepprivacy Oursmentioning

confidence: 99%

See 1 more Smart Citation

Attribute-preserving Face Dataset Anonymization via Latent Code Optimization

Barattin¹,

Tzelepis²,

Patras³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…LAION-5B's scale enables novel dataset curation for computer vision related tasks. Recently, researchers have utilized both LAION-5B and a subset, LAION-400M, as a data source in vision related tasks such as facial representation learning [96] and invasive species mitigation [38]. Within LAION, we have compiled from LAION-5B both LAION-High-Resolution 6 , a 170M subset for superresolution models, and LAION-Aesthetic…”

Section: Usage Examplesmentioning

confidence: 99%

LAION-5B: An open large-scale dataset for training next generation image-text models

Schuhmann¹,

Beaumont²,

Vencu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Groundbreaking language-vision architectures like CLIP and DALL-E proved the utility of training on large amounts of noisy image-text data, without relying on expensive accurate labels used in standard vision unimodal supervised learning. The resulting models showed capabilities of strong text-guided image generation and transfer to downstream tasks, while performing remarkably at zero-shot classification with noteworthy out-of-distribution robustness. Since then, large-scale language-vision models like ALIGN, BASIC, GLIDE, Flamingo and Imagen made further improvements. Studying the training and capabilities of such models requires datasets containing billions of image-text pairs. Until now, no datasets of this size have been made openly available for the broader research community. To address this problem and democratize research on large-scale multi-modal models, we present LAION-5B -a dataset consisting of 5.85 billion CLIP-filtered image-text pairs, of which 2.32B contain English language. We show successful replication and fine-tuning of foundational models like CLIP, GLIDE and Stable Diffusion using the dataset, and discuss further experiments enabled with an openly available dataset of this scale. Additionally we provide several nearest neighbor indices, an improved web-interface for dataset exploration and subset generation, and detection scores for watermark, NSFW, and toxic content detection. 1 1 Project page: https://laion.ai/laion-5b-a-new-era-of-open-large-scale-multi-modal-datasets/

show abstract

“…Third, we apply FaRL for Facial Representation Learning (FaRL) [90] which is a pretrained Transformer model intended for tasks related to facial features. Using the FaRL image encoder we obtain 512-dimensional features for the extracted face frames.…”

Section: Videomentioning

confidence: 99%

Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results

Christ¹,

Amiriparian²,

Kathan³

et al. 2022

Preprint

View full text Add to dashboard Cite

Humour is a substantial element of human affect and cognition. Its automatic understanding can facilitate a more naturalistic human-device interaction and the humanisation of artificial intelligence. Current methods of humour detection are solely based on staged data making them inadequate for 'real-world' applications. We address this deficiency by introducing the novel Passau-Spontaneous Football Coach Humour (Passau-SFCH) dataset, comprising of about 11 hours of recordings. The Passau-SFCH dataset is annotated for the presence of humour and its dimensions (sentiment and direction) as proposed in Martin's Humor Style Questionnaire. We conduct a series of experiments, employing pretrained Transformers, convolutional neural networks, and expert-designed features. The performance of each modality (text, audio, video) for spontaneous humour recognition is analysed and their complementarity is investigated. Our findings suggest that for the automatic analysis of humour and its sentiment, facial expressions are most promising, while humour direction can be best modelled via text-based features. The results reveal considerable differences among various subjects, highlighting the individuality of humour usage and style. Further, we observe that a decision-level fusion yields the best recognition result. Finally, we make our code publicly available at https://www.github.com/EIHW/passau-sfch. The Passau-SFCH dataset is available upon request.

show abstract

General Facial Representation Learning in a Visual-Linguistic Manner

Cited by 62 publications

References 57 publications

Attribute-preserving Face Dataset Anonymization via Latent Code Optimization

Attribute-preserving Face Dataset Anonymization via Latent Code Optimization

LAION-5B: An open large-scale dataset for training next generation image-text models

Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results

Contact Info

Product

Resources

About