Video2mesh: 3D human pose and shape recovery by a temporal convolutional transformer network

Abstract

From a 2D video of a person in action, human mesh recovery aims to infer the 3D human pose and shape frame by frame. Despite progress on video-based human pose and shape estimation, it is still challenging to guarantee high accuracy and smoothness simultaneously. To tackle this problem, we propose a Video2mesh, a temporal convolutional transformer (TConvTransformer) based temporal network which is able to recover accurate and smooth human mesh from 2D video. The temporal convolution block achieves the sequence-level smoothness by aggregating image features from adjacent frames. The subsequent multi-attention transformer improves the accuracy due to its multi-subspace for better middle-frame feature representation. Meanwhile, we add a TConvTransformer discriminator which is trained together with our 3D human mesh temporal encoder. This TConvTransformer discriminator further improves the accuracy and smoothness by restricting the pose and shape in a more reliable space based on the AMASS dataset. We conduct extensive experiments on three standard benchmark datasets and show that our proposed Video2mesh outperforms other state-of-the-art methods in both accuracy and smoothness.

Graph-based Semi-supervised Learning for 3D Pose Estimation

“…The poses are shifted to a common origin point and are rescaled to the same size which makes our model independent of the subject size and position in the frame. The temporal information is exploited using the LSTM network [17]. Though FCNs suffer from vanishing gradient problem, this problem can be eliminated by using residual connections in the network.…”

“…3. Readout: Following the message passing step, the nal node representations nd application in downstream tasks, such as node classi cation, link prediction, or graph-level prediction [17].…”