High‐Resolution Neural Face Swapping for Visual Effects

Abstract

In this paper, we propose an algorithm for fully automatic neural face swapping in images and videos. To the best of our knowledge, this is the first method capable of rendering photo‐realistic and temporally coherent results at megapixel resolution. To this end, we introduce a progressively trained multi‐way comb network and a light‐ and contrast‐preserving blending method. We also show that while progressive training enables generation of high‐resolution images, extending the architecture and training data beyond two people allows us to achieve higher fidelity in generated expressions. When compositing the generated expression onto the target face, we show how to adapt the blending strategy to preserve contrast and low‐frequency lighting. Finally, we incorporate a refinement strategy into the face landmark stabilization algorithm to achieve temporal stability, which is crucial for working with high‐resolution videos. We conduct an extensive ablation study to show the influence of our design choices on the quality of the swap and compare our work with popular state‐of‐the‐art methods.

The digital resurrection of Margaret Thatcher: Creative, technological and legal dilemmas in the use of deepfakes in screen drama

“…These used an encoder network combined with two decoder networks; the shared encoder network encodes source and target faces into a shared latent space (Liu et al, 2017), and the two decoder networks reconstruct the source and target images from the latent representation. The approach by Naruniec et al (2020) both generalized the number of outputs, and utilized high resolution inputs and outputs to lead to film quality face swapping. However, all these approaches reply on a large volume of source and target data to produce viable results.…”

Beyond deep fakes: Conceptual framework, applications, and research agenda for neural rendering of realistic digital faces

“…The approach that has most commonly been used for high quality face swapping is image-to-image translation using conditional GANs [14], where training data is provided for the target face and the source face. To place a source face onto a target face the system ideally requires adequate training data of both faces in similar poses and lighting.…”

“…The original Deepfake program uses 3D morphable models. Deepfakelabs approach uses a dual Y-shaped autoencoder implementation architecture [14].…”

Uso y divulgación de la imagen personal: enfoques en el derecho romano, en el derecho colombiano y su actual interacción con la Inteligencia Artificial

“…3, 2010. Available at ssrn: https://ssrn.com/abstract=1458637 uso y divulgación de la imagen personal: enfoques en el derecho romano, en el derecho colombiano y su actual interacción con la inteligencia artificial con alrededor de 125 videos interactivos en los que de forma hiperrealista aparece el artista y brinda información sobre su obra e interactúa con los visitantes 74 ; también Disney anunció recientemente que han desarrollado un algoritmo de intercambio automático de caras en video y fotografía que, dada su sofisticación e hiperrealismo, va a ser en el corto plazo una herramienta de efectos visuales para sus películas 75 , lo que hará que se difunda su uso en la producción de cine y televisión.…”