Putting People in their Place: Monocular Regression of 3D People in Depth

Sun, Yu; Liu, Wu; Bao, Qing; Fu, Yili; Mei, Tao; Black, Michael J.

doi:10.1109/cvpr52688.2022.01289

Cited by 102 publications

(63 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In early times, Han et al [4] proposed to compress the silhouettes of the sequence in the temporal dimension, thus obtaining Gait Energy Image (GEI). Recently, due to the success of deep learning on multimedia and computer vision tasks [5,20,21,27,33,[36][37][38][39][40], deep learning-based methods also dominated the performance of gait recognition. For example, Shiraga et al [25] and Wu et al [32] proposed to learn gait features from GEIs by CNNs and significantly outperformed previous methods.…”

Section: Related Work 21 Gait Recognitionmentioning

confidence: 99%

Gait Recognition in the Wild with Multi-hop Temporal Switch

Zheng

Liu

et al. 2022

Proceedings of the 30th ACM International Conference on Multimedia

Self Cite

View full text Add to dashboard Cite

Existing studies for gait recognition are dominated by in-the-lab scenarios. Since people live in real-world senses, gait recognition in the wild is a more practical problem that has recently attracted the attention of the community of multimedia and computer vision. Current methods that obtain state-of-the-art performance on inthe-lab benchmarks achieve much worse accuracy on the recently proposed in-the-wild datasets because these methods can hardly model the varied temporal dynamics of gait sequences in unconstrained scenes. Therefore, this paper presents a novel multi-hop temporal switch method to achieve effective temporal modeling of gait patterns in real-world scenes. Concretely, we design a novel gait recognition network, named Multi-hop Temporal Switch Network (MTSGait), to learn spatial features and multi-scale temporal features simultaneously. Different from existing methods that use 3D convolutions for temporal modeling, our MTSGait models the temporal dynamics of gait sequences by 2D convolutions. By this means, it achieves high efficiency with fewer model parameters and reduces the difficulty in optimization compared with 3D convolution-based models. Based on the specific design of the 2D convolution kernels, our method can eliminate the misalignment of features among adjacent frames. In addition, a new sampling strategy, i.e., non-cyclic continuous sampling, is proposed to make the model learn more robust temporal features. Finally, the proposed method achieves * This work was done when Jinkai Zheng was an intern at JD Explore Academy.

show abstract

Section: Related Work 21 Gait Recognitionmentioning

confidence: 99%

Gait Recognition in the Wild with Multi-hop Temporal Switch

Zheng

Liu

et al. 2022

Proceedings of the 30th ACM International Conference on Multimedia

Self Cite

View full text Add to dashboard Cite

show abstract

“…Early methods are focused on retrieval-based pipelines to retarget the subject in a video to the desired motion with optical flow [11] or 3D human skeleton [41]. Recent methods usually adopt deep neural networks, especially GANs [13], and exploit 2D human pose [2,5,30,35,36], human parsing [8,21,26,39], 3D human pose [37], and other 3D information [20,22,31,44] for human motion transfer [23,24].…”

Section: Related Work 21 Human Motion Transfermentioning

confidence: 99%

Delving into the Frequency: Temporally Consistent Human Motion Transfer in the Fourier Space

Yang

Liu

et al. 2022

Proceedings of the 30th ACM International Conference on Multimedia

Self Cite

View full text Add to dashboard Cite

Human motion transfer refers to synthesizing photo-realistic and temporally coherent videos that enable one person to imitate the motion of others. However, current synthetic videos suffer from the temporal inconsistency in sequential frames that significantly degrades the video quality, yet is far from solved by existing methods in the pixel domain. Recently, some works on DeepFake detection try to distinguish the natural and synthetic images in the frequency domain because of the frequency insufficiency of image synthesizing methods. Nonetheless, there is no work to study the temporal inconsistency of synthetic videos from the aspects of the frequencydomain gap between natural and synthetic videos. Therefore, in this paper, we propose to delve into the frequency space for temporally consistent human motion transfer. First of all, we make the first comprehensive analysis of natural and synthetic videos in the frequency domain to reveal the frequency gap in both the spatial dimension of individual frames and the temporal dimension of the video. To close the frequency gap between the natural and synthetic videos, we propose a novel Frequency-based human MOtion TRansfer framework, named FreMOTR, which can effectively mitigate the spatial artifacts and the temporal inconsistency of the synthesized videos. FreMOTR explores two novel frequency-based regularization modules: 1) the Frequency-domain Appearance Regularization (FAR) to improve the appearance of the person in individual frames and 2) Temporal Frequency Regularization (TFR) to guarantee the temporal consistency between adjacent frames. Finally, comprehensive experiments demonstrate that the FreMOTR not only yields superior performance in temporal consistency metrics but also improves the frame-level visual quality *This work was done when Guang Yang was an intern at JD Explore Academy. †Wu Liu and Juan Cao are corresponding authors.

show abstract

“…Although monocular human pose and shape estimation [15,44] has been extensively explored over the past years, estimating global space locations together with human poses and shapes for multiple people from a single image is still a difficult problem due to the depth ambiguity. Existing methods [13,37] reconstruct 3D poses, shapes and relative positions of the reconstructed human meshes by assuming a constant focal length. But the methods are limited to small scenes with a common FoV (Field of View).…”

Section: Introductionmentioning

confidence: 99%