Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection

Zolfaghari, Mehdi; Oliveira, Gabriel L.; Sedaghat, Nima; Brox, Thomas

doi:10.1109/iccv.2017.316

Cited by 205 publications

(146 citation statements)

References 48 publications

Supporting

Mentioning

144

Contrasting

Order By: Relevance

“…This result suggests that high-precision pose estimation is not essential within the STAR framework for the purpose of action recognition. This is in contrast to other pose-based approaches that saw significant performance improvements using ground-truth keypoint locations [33], [45], [46]. The inference speeds of each end-to-end architecture are also reported in Table I.…”

Section: B Star-net Resultsmentioning

confidence: 96%

“…On UTD-MHAD, the effect of combining the models decreased performance. The results in Table II also show that STAR-Net outperforms several methods using richer data modalities, but surrenders 2.8% to the state-of-the-art pose-based method of Liu et al [32], who used spatial rank pooling to encode the evolution of 2D [33] 57.0 Chron et al (P-CNN) [45] 61.1 Gkioxari et al (Action Tubes) [49] 62.5 Peng et al (MR TS R-CNN) [10] 71.1 Zolfaghari et al (Chained) [46] 76.1 STAR-Net 64.3 pose images and averaged pose heatmaps (i.e., as separate streams). Interestingly, the performance of each stream alone was 85.6% and 74.9%, respectively, indicating that these streams were highly complementary.…”

Section: Comparison With the State-of-the-artmentioning

confidence: 99%

See 1 more Smart Citation

STAR-Net: Action Recognition using Spatio-Temporal Activation Reprojection

McNally

Wong

McPhee

2019

2019 16th Conference on Computer and Robot Vision (CRV)

View full text Add to dashboard Cite

While depth cameras and inertial sensors have been frequently leveraged for human action recognition, these sensing modalities are impractical in many scenarios where cost or environmental constraints prohibit their use. As such, there has been recent interest on human action recognition using low-cost, readily-available RGB cameras via deep convolutional neural networks. However, many of the deep convolutional neural networks proposed for action recognition thus far have relied heavily on learning global appearance cues directly from imaging data, resulting in highly complex network architectures that are computationally expensive and difficult to train. Motivated to reduce network complexity and achieve higher performance, we introduce the concept of spatiotemporal activation reprojection (STAR). More specifically, we reproject the spatio-temporal activations generated by human pose estimation layers in space and time using a stack of 3D convolutions. Experimental results on UTD-MHAD and J-HMDB demonstrate that an end-to-end architecture based on the proposed STAR framework (which we nickname STAR-Net) is proficient in single-environment and small-scale applications. On UTD-MHAD, STAR-Net outperforms several methods using richer data modalities such as depth and inertial sensors.

show abstract

Section: B Star-net Resultsmentioning

confidence: 96%

Section: Comparison With the State-of-the-artmentioning

confidence: 99%

STAR-Net: Action Recognition using Spatio-Temporal Activation Reprojection

McNally

Wong

McPhee

2019

2019 16th Conference on Computer and Robot Vision (CRV)

View full text Add to dashboard Cite

show abstract

“…In addition, since DD-net employs one-dimensional CNNs to extract the feature, it is much faster than other models that use RNNs [31], [22], [32], [25] or 2D/3D CNNs [5], [39], [7], [8], [28]. During its inferences, DD-Net's speed can reach around 3,500 FPS on one GPU (i.e., GTX 1080Ti), or, 2,000 FPS on one CPU (i.e., Intel E5-2620).…”

Section: Results Analysis and Discussionmentioning

confidence: 99%

“…For skeleton-based action recognition, two types of input features are commonly used: the geometric feature [18], [22] and the Cartesian coordinate feature [31], [32], [34], [6], [7]. The Cartesian coordinate feature is variant to locations and viewpoints.…”

Section: A Modeling Location-viewpoint Invariant Feature By Joint Comentioning

confidence: 99%

Make Skeleton-based Action Recognition Model Smaller, Faster and Better

Yang

Wu²,

Sakti

et al. 2019

Proceedings of the ACM Multimedia Asia

115

View full text Add to dashboard Cite

Although skeleton-based action recognition has achieved great success in recent years, most of the existing methods may suffer from a large model size and slow execution speed. To alleviate this issue, we analyze skeleton sequence properties to propose a Double-feature Double-motion Network (DD-Net) for skeleton-based action recognition. By using a lightweight network structure (i.e., 0.15 million parameters), DD-Net can reach a super fast speed, as 3,500 FPS on one GPU, or, 2,000 FPS on one CPU. By employing robust features, DD-Net achieves the state-of-the-art performance on our experiment datasets: SHREC (i.e., hand actions) and JHMDB (i.e., body actions). Our code is on https://github.com/fandulu/DD-Net.

show abstract

“…Liu et al [44] jointly learned the regression and classification network with multi-modal data for action detection. Though the methods in [14], [15], [44], [45] illustrate that the introduction of multiple cues improve the performance for action analytics, they are limited by the strict data requirements. [15] requires aligned skeletons and depth data in training.…”

Section: B Skeleton-based Action Recognitionmentioning

confidence: 99%

Modality Compensation Network: Cross-Modal Adaptation for Action Recognition

Song

Liu

et al. 2020

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

With the prevalence of RGB-D cameras, multimodal video data have become more available for human action recognition. One main challenge for this task lies in how to effectively leverage their complementary information. In this work, we propose a Modality Compensation Network (MCN) to explore the relationships of different modalities, and boost the representations for human action recognition. We regard RGB/optical flow videos as source modalities, skeletons as auxiliary modality. Our goal is to extract more discriminative features from source modalities, with the help of auxiliary modality. Built on deep Convolutional Neural Networks (CNN) and Long Short Term Memory (LSTM) networks, our model bridges data from source and auxiliary modalities by a modality adaptation block to achieve adaptive representation learning, that the network learns to compensate for the loss of skeletons at test time and even at training time. We explore multiple adaptation schemes to narrow the distance between source and auxiliary modal distributions from different levels, according to the alignment of source and auxiliary data in training. In addition, skeletons are only required in the training phase. Our model is able to improve the recognition performance with source data when testing. Experimental results reveal that MCN outperforms stateof-the-art approaches on four widely-used action recognition benchmarks.

show abstract

Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection

Cited by 205 publications

References 48 publications

STAR-Net: Action Recognition using Spatio-Temporal Activation Reprojection

STAR-Net: Action Recognition using Spatio-Temporal Activation Reprojection

Make Skeleton-based Action Recognition Model Smaller, Faster and Better

Modality Compensation Network: Cross-Modal Adaptation for Action Recognition

Contact Info

Product

Resources

About