Cross-Identity Motion Transfer for Arbitrary Objects Through Pose-Attentive Video Reassembling

Jeon, Subin; Nam, Seonghyeon; Oh, Seoung Wug; Kim, Seon Joo

doi:10.1007/978-3-030-58586-0_18

Cited by 7 publications

(8 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The model-free approaches [30,31,19,32,34] does not rely on pre-trained third-party models, and extend the model-based method to arbitrary objects. Aliaksandr et al [30] proposed a model-free motion transfer model Monky-Net that can apply motion transfer on arbitrary objects with an unsupervised key point detector trained by reconstruction loss [18].…”

Section: Related Workmentioning

confidence: 99%

“…A generator module is utilized to generate final result with the warped source image feature. Subin et al [19] proposed pose attention mechanism with an unsupervised key point detector to model motion. Recently, Aliaksandr et al [32] improved FOMM with an advanced motion model and background motion model to MRAA.…”

Section: Related Workmentioning

confidence: 99%

“…To address this issue, we build a cyclic reconstruction framework inspired by the CycleGAN [42] and cross-identity [19]. As shown in the right-hand side of Fig 2 , we employ two basic basic MT models that share the same parameters.…”

Section: Overviewmentioning

confidence: 99%

“…However, such training mode cannot be directly applied to the cross-domain motion transfer because no ground-truth is available. In our proposed MAA approach, we build a cyclic reconstruction pipeline inspired by CycleGAN [42] and cross-identity [19]. In particular, given a source image and a driving frame obtained from different domains, we first obtain a synthesized image using a basic motion transfer (MT) model, e.g., the model in [31] or [32].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Motion and Appearance Adaptation for Cross-Domain Motion Transfer

Xu¹,

Wang²,

Deng³

et al. 2022

Preprint

View full text Add to dashboard Cite

Motion transfer aims to transfer the motion of a driving video to a source image. When there are considerable differences between object in the driving video and that in the source image, traditional single domain motion transfer approaches often produce notable artifacts; for example, the synthesized image may fail to preserve the human shape of the source image (cf . Fig. 1 (a)). To address this issue, in this work, we propose a Motion and Appearance Adaptation (MAA) approach for cross-domain motion transfer, in which we regularize the object in the synthesized image to capture the motion of the object in the driving frame, while still preserving the shape and appearance of the object in the source image. On one hand, considering the object shapes of the synthesized image and the driving frame might be different, we design a shape-invariant motion adaptation module that enforces the consistency of the angles of object parts in two images to capture the motion information. On the other hand, we introduce a structure-guided appearance consistency module designed to regularize the similarity between the corresponding patches of the synthesized image and the source image without affecting the learned motion in the synthesized image. Our proposed MAA model can be trained in an end-to-end manner with a cyclic reconstruction loss, and ultimately produces a satisfactory motion transfer result (cf . Fig. 1 (b)). We conduct extensive experiments on human dancing dataset Mixamo-Video to Fashion-Video and human face dataset Vox-Celeb to Cufs; on both of these, our MAA model outperforms existing methods both quantitatively and qualitatively.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Overviewmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Motion and Appearance Adaptation for Cross-Domain Motion Transfer

Xu¹,

Wang²,

Deng³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…To overcome the huge data requirement of individual subjects and improve the generalization of personalized methods, recent works mainly focus on general-purpose methods [5,12,15,25,39,45,50]. This type of method aims to learn a model that can be adapted for the generation of unseen persons.…”

Section: Introductionmentioning

confidence: 99%

REMOT: A Region-to-Whole Framework for Realistic Human Motion Transfer

Yang

Liu

et al. 2022

Proceedings of the 30th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Human Video Motion Transfer (HVMT) aims to, given an image of a source person, generate his/her video that imitates the motion of the driving person. Existing methods for HVMT mainly exploit Generative Adversarial Networks (GANs) to perform the warping operation based on the flow estimated from the source person image and each driving video frame. However, these methods always generate obvious artifacts due to the dramatic differences in poses, scales, and shifts between the source person and the driving person. To overcome these challenges, this paper presents a novel REgionto-whole human MOtion Transfer (REMOT) framework based on GANs. To generate realistic motions, the REMOT adopts a progressive generation paradigm: it first generates each body part in the driving pose without flow-based warping, then composites all parts into a complete person of the driving motion. Moreover, to preserve the natural global appearance, we design a Global Alignment Module to align the scale and position of the source person with those of the driving person based on their layouts. Furthermore, we propose a Texture Alignment Module to keep each part of the person aligned according to the similarity of the texture. Finally, through extensive quantitative and qualitative experiments, our REMOT achieves state-of-the-art results on two public benchmarks.

show abstract

OOOE: Only-One-Object-Exists Assumption to Find Very Small Objects in Chest Radiographs

Nam¹,

Kim²,

Lee³

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The accurate localization of inserted medical tubes and parts of human anatomy is a common problem when analyzing chest radiographs and something deep neural networks could potentially automate. However, many foreign objects like tubes and various anatomical structures are small in comparison to the entire chest X-ray, which leads to severely unbalanced data and makes training deep neural networks difficult. In this paper, we present a simple yet effective 'Only-One-Object-Exists' (OOOE) assumption to improve the deep network's ability to localize small landmarks in chest radiographs. The OOOE enables us to recast the localization problem as a classification problem and we can replace commonly used continuous regression techniques with a multi-class discrete objective. We validate our approach using a large scale proprietary dataset of over 100K radiographs as well as publicly available RANZCR-CLiP Kaggle Challenge dataset and show that our method consistently outperforms commonly used regression-based detection models as well as commonly used pixel-wise classification methods. Additionally, we find that the method using the OOOE assumption generalizes to multiple detection problems in chest X-rays and the resulting model shows state-of-the-art performance on detecting various tube tips inserted to the patient as well as patient anatomy.

show abstract

Cross-Identity Motion Transfer for Arbitrary Objects Through Pose-Attentive Video Reassembling

Cited by 7 publications

References 23 publications

Motion and Appearance Adaptation for Cross-Domain Motion Transfer

Motion and Appearance Adaptation for Cross-Domain Motion Transfer

REMOT: A Region-to-Whole Framework for Realistic Human Motion Transfer

OOOE: Only-One-Object-Exists Assumption to Find Very Small Objects in Chest Radiographs

Contact Info

Product

Resources

About