The objective of this paper is a neural network model that controls the pose and expression of a given face, using another face or modality (e.g. audio). This model can then be used for lightweight, sophisticated video and image editing. We make the following three contributions. First, we introduce a network, X2Face, that can control a source face (specified by one or more frames) using another face in a driving frame to produce a generated frame with the identity of the source frame but the pose and expression of the face in the driving frame. Second, we propose a method for training the network fully self-supervised using a large collection of video data. Third, we show that the generation process can be driven by other modalities, such as audio or pose codes, without any further training of the network. The generation results for driving a face with another face are compared to state-of-the-art self-supervised/supervised methods. We show that our approach is more robust than other methods, as it makes fewer assumptions about the input data. We also show examples of using our framework for video face editing.* Denotes equal contribution.The source face is instantiated from a single or multiple source frames, which are extracted from the same face track. The driving vector may come from multiple modalities: a driving frame from the same or another video face track, pose information, or audio information; this is illustrated in Fig. 1. The generated frame resulting from X2Face has the identity, hairstyle, etc. of the source face but the properties of the driving vector (e.g. the given pose, if pose information is given; or the driving frame's expression/pose, if a driving frame is given). The network is trained in a self-supervised manner using pairs of source and driving frames. These frames are input to two subnetworks: the embedding network and the driving network (see Fig. 2). By controlling the information flow in the network architecture, the model learns to factorise the problem. The embedding network learns an embedded face representation for the source face -effectively face frontalisation; the driving network learns how to map from this embedded face representation to the generated frame via an embedding, named the driving vector. The X2Face network architecture is described in Section 3.1, and the self-supervised training framework in Section 3.2. In addition we make two further contributions. First, we propose a method for linearly regressing from a set of labels (e.g. for head pose) or features (e.g. from audio) to the driving vector; this is described in Section 4. The performance is evaluated in Section 5, where we show (i) the robustness of the generated results compared to state-of-the-art self-supervised [45] and supervised [1] methods; and (ii) the controllability of the network using other modalities, such as audio or pose. The second contribution, described in Section 6, shows how the embedded face representation can be used for video face editing, e.g. adding facial decorations in the manne...

Figure 1: End-to-end view synthesis. Given a single RGB image (red), SynSin generates images of the scene at new viewpoints (blue). SynSin predicts a 3D point cloud, which is projected onto new views using our differentiable renderer; the rendered point cloud is passed to a GAN to synthesise the output image. SynSin is trained end-to-end, without 3D supervision.

The objective of this paper is 3D shape understanding from single and multiple images. To this end, we introduce a new deep-learning architecture and loss function, Sil-Net, that can handle multiple views in an order-agnostic manner. The architecture is fully convolutional, and for training we use a proxy task of silhouette prediction, rather than directly learning a mapping from 2D images to 3D shape as has been the target in most recent work.We demonstrate that with the SilNet architecture there is generalisation over the number of views -for example, SilNet trained on 2 views can be used with 3 or 4 views at test-time; and performance improves with more views.We introduce two new synthetics datasets: a blobby object dataset useful for pretraining, and a challenging and realistic sculpture dataset; and demonstrate on these datasets that SilNet has indeed learnt 3D shape. Finally, we show that SilNet exceeds the state of the art on the ShapeNet benchmark dataset [6] at generating silhouettes in new viewpoints, and we use SilNet to generate novel views of the sculpture dataset.

This work explores how to use self-supervised learning on videos to learn a class-specific image embedding that encodes pose and shape information. At train time, two frames of the same video of an object class (e.g. human upper body) are extracted and each encoded to an embedding. Conditioned on these embeddings, the decoder network is tasked to transform one frame into another. To successfully perform long range transformations (e.g. a wrist lowered in one image should be mapped to the same wrist raised in another), we introduce a hierarchical probabilistic network decoder model. Once trained, the embedding can be used for a variety of downstream tasks and domains. We demonstrate our approach quantitatively on three distinct deformable object classeshuman full bodies, upper bodies, faces -and show experimentally that the learned embeddings do indeed generalise. They achieve state-ofthe-art performance in comparison to other self-supervised methods trained on the same datasets, and approach the performance of fully supervised methods.