LSTM Pose Machines

Abstract

We observed that recent state-of-the-art results on single image human pose estimation were achieved by multistage Convolution Neural Networks (CNN). Notwithstanding the superior performance on static images, the application of these models on videos is not only computationally intensive, it also suffers from performance degeneration and flicking. Such suboptimal results are mainly attributed to the inability of imposing sequential geometric consistency, handling severe image quality degradation (e.g. motion blur and occlusion) as well as the inability of capturing the temporal correlation among video frames. In this paper, we proposed a novel recurrent network to tackle these problems. We showed that if we were to impose the weight sharing scheme to the multi-stage CNN, it could be re-written as a Recurrent Neural Network (RNN). This property decouples the relationship among multiple network stages and results in significantly faster speed in invoking the network for videos. It also enables the adoption of Long Short-Term Memory (LSTM) units between video frames. We found such memory augmented RNN is very effective in imposing geometric consistency among frames. It also well handles input quality degradation in videos while successfully stabilizes the sequential outputs. The experiments showed that our approach significantly outperformed current state-ofthe-art methods on two large-scale video pose estimation benchmarks. We also explored the memory cells inside the LSTM and provided insights on why such mechanism would benefit the prediction for video-based pose estimations. 1

Dynamic Kernel Distillation for Efficient Pose Estimation in Videos

“…In [25], Song et al propose a Thin-Slicing network that uses dense optical flow to warp and align heatmaps of neighboring frames and then performs spatial-temporal inference via message passing through the graph constructed by joint candidates and their relationships among aligned heatmaps. [11] and [20] sequentially estimate human poses in videos following the Encoder-RNN-Decoder framework. Given a frame, this kind of framework first uses an encoder network to learn high-level image representations, then RNN units to explicitly propagate temporal information between neighboring frames and produce hidden states, and finally a decoder network to take hidden states as input and output pose estimation results of current frame.…”

“…Its distilled pose kernels can be applied to fast localize body joints with simple convolution, further improving the efficiency. In addition, it can directly leverage temporal cues of one frame to assist body joint localization in the following frame, without requiring auxiliary optical flow models [25] or decoders appended to RNN units [20]. It can also fast distill pose kernels in a one-shot manner, avoiding complex iterating utilized by previous online kernel learning models [4,27].…”

“…3 show the comparisons of our DKD model with state-of-the-arts on Penn Action dataset. In particular, the method proposed in [20] follows the Encoder-RNNs-Decoder framework with Convolutional LSTMs, while [25] exploits optical flow models to align confidence maps of neighboring frames. We report the performance of our model with both person and torso size as reference distance under the PCK evaluation metric.…”

“…Compared to its still-image based counterpart, this task is more challenging due to its low-latency requirement and various distracting factors, e.g., motion blur, pose variation and viewpoint change. Prior CNN based methods to solve this task [10,22,25,20] usually use a large network to extract representative features for every frame and localize body joints based on them via pixel-wise classification. Some recent works also incorporate temporal cues from optical flow [9] or RNN units [30] to improve the performance, as shown in Fig.…”

“…We reproduce the results of[20] with PCK normalized by torso size via running the codes released by the authors on the repo: https://github.com/lawy623/LSTM Pose Machines. The running time is counted on GPU GTX 1080ti for both[20] and our model.…”