Temporally Consistent Semantic Video Editing

DisCO: Portrait Distortion Correction with Perspective-Aware 3D GANs

“…Other methods combine the optimization and encoder approaches and propose hybrid strategies by using the encoder for initialization and refining the latent code by optimization [35,95]. Recent 2D GAN inversion methods achieve faithful reconstruction with high editing capabilities and have been extended for video editing [2,75,88]. However, editing 3D-related attributes such as camera parameters and head pose remains inconsistent and prone to severe flickering as the pre-trained generator is unaware of the 3D structure.…”

MEGANE: Morphable Eyeglass and Avatar Network

“…Generative models have demonstrated remarkable ability in synthesizing photorealistic images, including human faces [27]. Recent work has extended these models to add intuitive semantic editing, such as synthesis of glasses on faces [20,30,35,70,73]. Fader Networks [30] disentangle the salient image information, and then generate different images by varying attribute values, including glasses on faces.…”

“…Subsequent work has proposed two decoders for modeling latent representations and facial attributes [20], selective transfer units [35], and geometryaware flow [76] to further improve editing fidelity. Yao et al [73] extend facial attribute editing to video sequences via latent transformation and a identity preservation loss, which is further improved by Xu et al [70], incorporating flow-based consistency. More recent works propose 3Daware generative models to achieve view-consistent synthesis [8,10,49,55,63,67,71].…”

“…VideoEditGAN [70] is a SOTA image-based editing method that allows us to insert glasses on face images. As shown in Fig.…”

“…Another group of approaches aims to synthesize the composition of glasses in the image domain [30,70,73] by leveraging powerful 2D generative models [27]. While these approaches can produce photorealistic images, animation results typically suffer from view and temporal inconsistencies due to the lack of 3D information.…”