Fast Face-Swap Using Convolutional Neural Networks

Abstract

We consider the problem of face swapping in images, where an input identity is transformed into a target identity while preserving pose, facial expression and lighting. To perform this mapping, we use convolutional neural networks trained to capture the appearance of the target identity from an unstructured collection of his/her photographs. This approach is enabled by framing the face swapping problem in terms of style transfer, where the goal is to render an image in the style of another one. Building on recent advances in this area, we devise a new loss function that enables the network to produce highly photorealistic results. By combining neural networks with simple pre-and post-processing steps, we aim at making face swap work in real-time with no input from the user.

Mask-Guided Portrait Editing With Conditional GANs

“…With the proposed structure that disentangles and recombines facial instance embeddings with face masks, our method also enhances over face swapping methods [5,26] by supporting explicit face and hair swapping.…”

Live Face De-Identification in Video

“…While these classical face swapping methods work in the pixel space and copy the expression of the target image, recent deep-learning based work swaps the identity, while maintaining the other aspects of the source image [23]. In comparison to our work, [23] requires training a new network for every target person, the transferred expression does not show subtleties (which would be critical, e.g., for a speaking person), and the results are not as natural as ours. These limitations are probably a result of capturing the appearance of the target, by restricting the output to be similar, patch by patch, to a collection of patches from the target person.…”

“…These limitations are probably a result of capturing the appearance of the target, by restricting the output to be similar, patch by patch, to a collection of patches from the target person. Moreover, [23] is limited to stills and was not demonstrated on video.…”

FSNet: An Identity-Aware Generative Model for Image-Based Face Swapping

“…A similar technique was used in "FakeApp" [17], an easy-to-use application software for image-based face swapping using a DNN. Korshunova et al [11] considered face identities as artistic styles in neural style transfer [18], and performed face swapping by fine-tuning the pre-trained network using a dozens of images of an individual. Although these approaches facilitated the use of deep-learning techniques for face swapping, they share a common problem in that the users must prepare multiple input images of an individual.…”

“…To respond to the potential demand for creating more attractive face images with such photo-retouching software, many studies have been introduced for a range of applications including face image analysis [1][2][3][4] and manipulation [5][6][7][8][9] in computer vision and graphics communities. In such applications, face swapping is an important technique owing to its broad applications such as photomontage [5], virtual hairstyle fitting [9], privacy protection [6,10,11], and data augmentation for machine learning [12][13][14].…”