C2F-FWN: Coarse-to-Fine Flow Warping Network for Spatial-Temporal Consistent Motion Transfer

Abstract

Human video motion transfer (HVMT) aims to synthesize videos that one person imitates other persons' actions. Although existing GAN-based HVMT methods have achieved great success, they either fail to preserve appearance details due to the loss of spatial consistency between synthesized and exemplary images, or generate incoherent video results due to the lack of temporal consistency among video frames. In this paper, we propose Coarse-to-Fine Flow Warping Network (C2F-FWN) for spatial-temporal consistent HVMT. Particularly, C2F-FWN utilizes coarse-to-fine flow warping and Layout-Constrained Deformable Convolution (LC-DConv) to improve spatial consistency, and employs Flow Temporal Consistency (FTC) Loss to enhance temporal consistency. In addition, provided with multi-source appearance inputs, C2F-FWN can support appearance attribute editing with great flexibility and efficiency. Besides public datasets, we also collected a large-scale HVMT dataset named SoloDance for evaluation. Extensive experiments conducted on our SoloDance dataset and the iPER dataset show that our approach outperforms state-of-art HVMT methods in terms of both spatial and temporal consistency. Source code and the SoloDance dataset are available at https://github.com/wswdx/C2F-FWN.

REMOT: A Region-to-Whole Framework for Realistic Human Motion Transfer

“…• A Texture Alignment Module is proposed to align the features of the source image and the initially generated image to preserve more details like textures of clothes and edges of the bodies. We conduct extensive experiments on the iPER Dataset [25] and SoloDance Dataset [50]. Experimental results show that our model achieves the state-of-the-art both quantitatively and qualitatively.…”

“…However, the SMPL model [32] was only suitable for smooth human bodies, it cannot represent the human body with complex clothes. Different from warping the feature of the source image, C2F [50] estimated the optical flow of the clothing regions and directly warped the clothing region according to optical flow. Unfortunately, when the driving pose is greatly different from the source pose, such methods usually failed due to inaccurate flow estimations.…”

“…The task of Human video Motion Transfer (HVMT) is to synthesize a realistic video where a source person performs desired driving motion [33,50]. Specifically, given the source image and a driving video, it needs to make the person in the source image imitate the action of the person in the driving video.…”

“…To overcome the huge data requirement of individual subjects and improve the generalization of personalized methods, recent works mainly focus on general-purpose methods [5,12,15,25,39,45,50]. This type of method aims to learn a model that can be adapted for the generation of unseen persons.…”

“…Second, the drastic variations of poses and viewpoints between the source person and the driving person bring great challenges to human motion transfer. Some recent methods adopt warping flows, such as the optical flow [50], the transform flow [25], etc., to achieve pose transfer, as shown in Figure 1 (b). Nevertheless, the generated frames still have significant artifacts caused by the errors of flow estimation.…”