PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering

3D GAN Inversion for Controllable Portrait Image Animation

“…We present results of the proposed method with comparisons to other baseline methods, including PIRenderer [Ren et al 2021], a SoTA approach for portrait image animation, and a modified version of the proposed method using a 2D GAN. PIRenderer uses a pretrained 3DMM encoder to predict the expressions of the target frame and transfers them to the source image using a warping field predicted with a convolutional network.…”

“…Autoencoder networks, however, make it difficult to attain intuitive parametric control of the animation. Recent approaches aim to overcome this challenge using a keypointdriven motion flow field ] or a 3DMM [Ren et al 2021] to drive the portrait image edits. Yet, all these network architectures are oblivious to the underlying 3D structure of the content, making it challenging to produce photorealistic outputs across a range of viewpoints.…”

“…We evaluate the proposed method for portrait image animation and compare against the PIRenderer baseline [Ren et al 2021] and the proposed approach using a pre-trained 2D GAN [Karras et al 2020].…”

“…We observe similar trends in the quantitative results, reported in Table 1. These results are calculated over a dataset of 500 images from CelebA-HQ [Karras et al 2018] and 500 target video sequences sampled from VoxCeleb [Chung et al 2018; We compare the proposed approach with PIRenderer [Ren et al 2021] and the proposed approach using a pre-trained 2D GAN [Karras et al 2020] instead of the 3D GAN. Since we cannot directly control the pose with the 2D GAN, we only edit the expressions.…”

“…Moreover, these methods cannot usually be directly to portrait image animation since they usually require multiple images or a video sequence of a subject in order to synthesize a complete texture map. Finally, another class of methods animates portrait images by warping an input image based on a desired pose and expression and refining the output using a neural network [Ren et al 2021;Siarohin et al 2019;Zakharov et al 2020]. While these methods produce plausible results, they do not achieve the photorealistic quality of GAN-based methods, and generally have view inconsistencies for large deviations from the initial portrait image.…”