PSGAN++: Robust Detail-Preserving Makeup Transfer and Removal

Abstract

In this paper, we address the makeup transfer and removal tasks simultaneously, which aim to transfer the makeup from a reference image to a source image and remove the makeup from the with-makeup image respectively. Existing methods have achieved much advancement in constrained scenarios, but it is still very challenging for them to transfer makeup between images with large pose and expression differences, or handle makeup details like blush on cheeks or highlight on the nose. In addition, they are hardly able to control the degree of makeup during transferring or to transfer a specified part in the input face. These defects limit the application of previous makeup transfer methods to real-world scenarios. In this work, we propose a Pose and expression robust Spatial-aware GAN (abbreviated as PSGAN++). PSGAN++ is capable of performing both detail-preserving makeup transfer and effective makeup removal. For makeup transfer, PSGAN++ uses a Makeup Distill Network (MDNet) to extract makeup information, which is embedded into spatial-aware makeup matrices. We also devise an Attentive Makeup Morphing (AMM) module that specifies how the makeup in the source image is morphed from the reference image, and a makeup detail loss to supervise the model within the selected makeup detail area. On the other hand, for makeup removal, PSGAN++ applies an Identity Distill Network (IDNet) to embed the identity information from with-makeup images into identity matrices. Finally, the obtained makeup/identity matrices are fed to a Style Transfer Network (STNet) that is able to edit the feature maps to achieve makeup transfer or removal. To evaluate the effectiveness of our PSGAN++, we collect a Makeup Transfer In the Wild (MT-Wild) dataset that contains images with diverse poses and expressions and a Makeup Transfer High-Resolution (MT-HR) dataset that contains high-resolution images. Experiments demonstrate that PSGAN++ not only achieves state-of-the-art results with fine makeup details even in cases of large pose/expression differences but also can perform partial or degree-controllable makeup transfer. Both the code and the newly collected datasets will be released at https://github.com/wtjiang98/PSGAN.

Makeup transfer: A review

“…To address the problem that most models have difficulty controlling makeup intensity, the authors of Ref. [67] propose a pose and expression robust spatially aware GAN (abbreviated as PSGAN++). PSGAN++ is capable of performing detail‐preserving makeup transfers and effective de‐makeup.…”

Personalized facial makeup transfer based on outline correspondence

“…In recent years, there have been many approaches based on generative adversarial networks. [1][2][3][4][5] BeautyGAN 1 was one of the first methods to use GAN for makeup transfer, which uses a pixel-level histogram loss to achieve instance-level makeup transfer. PSGAN 2 utilizes the face analysis map and facial coordinate points to construct the pixel-level correspondence between the source image and the reference image, so as to solve the problem of face misalignment between different head poses and facial expressions.…”

“…In recent years, there have been many approaches based on generative adversarial networks 1‐5 . BeautyGAN 1 was one of the first methods to use GAN for makeup transfer, which uses a pixel‐level histogram loss to achieve instance‐level makeup transfer.…”

“…In recent years, generative adversarial network (GAN) technology has been widely applied to various computer vision tasks due to its ability to produce high-quality realistic images. GAN-based facial makeup transfer methods [1][2][3][4][5] can significantly improve the makeup transfer effect. On the other hand, there are many kinds of makeup styles, including daily simple makeup, stage performance makeup and personalized extreme makeup.…”

ACGAN: Age‐compensated makeup transfer based on homologous continuity generative adversarial network model

“…However, it is able to achieve partial and shade-controllable makeup transfer in most cases. To address the problem that most models have difficulty in controlling the intensity of makeup, the authors propose a spatially aware GAN for pose and expression robustness (abbreviated as PSGAN++) [20]. PSGAN++ is able to perform detail-preserving makeup transformation and effective makeup removal.…”