Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition

Chen, Yuxin; Zhang, Ziqi; Yuan, Chunfeng; Li, Bing; Ding, Ying; Hu, Weiming

doi:10.1109/iccv48922.2021.01311

Cited by 340 publications

(161 citation statements)

References 23 publications

Supporting

Mentioning

159

Contrasting

Order By: Relevance

“…Swin simplifies the multiscale attention using shifted window, which might lead to less overfitting to noisy data. The visual transformer-based encoder structure shows better performance while comparing with GCN-, CNN-and skeleton-transformer based encoder as illustrated in Table V for one shot action recognition on NTU 120 dataset, where CTR-GCN [29] and ST-TR [28] are separately leveraged for GCN-based and skeleton-transformer based encoders. As shown in Table I, we also evaluate the performance of the proposed two-stage training strategy with a CNN-based encoder (specifically SL-DML), marked as PROCNN.…”

Section: B Experiments Resultsmentioning

confidence: 99%

ProFormer: Learning Data-efficient Representations of Body Movement with Prototype-based Feature Augmentation and Visual Transformers

Peng¹,

Roitberg²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Automatically understanding human behaviour allows household robots to identify the most critical needs and plan how to assist the human according to the current situation. However, the majority of such methods are developed under the assumption that a large amount of labelled training examples is available for all concepts-of-interest. Robots, on the other hand, operate in constantly changing unstructured environments, and need to adapt to novel action categories from very few samples. Methods for data-efficient recognition from body poses increasingly leverage skeleton sequences structured as image-like arrays and then used as input to convolutional neural networks. We look at this paradigm from the perspective of transformer networks, for the first time exploring visual transformers as data-efficient encoders of skeleton movement. In our pipeline, body pose sequences cast as image-like representations are converted into patch embeddings and then passed to a visual transformer backbone optimized with deep metric learning. Inspired by recent success of feature enhancement methods in semi-supervised learning, we further introduce PROFORMER -an improved training strategy which uses soft-attention applied on iteratively estimated action category PROtotypes used to augment the embeddings and compute an auxiliary consistency loss. Extensive experiments consistently demonstrate the effectiveness of our approach for one-shot recognition from body poses, achieving state-of-the-art results on multiple datasets and surpassing the best published approach on the challenging NTU-120 one-shot benchmark by 1.84%. Our code will be made publicly available at https: //github.com/KPeng9510/ProFormer.

show abstract

Section: B Experiments Resultsmentioning

confidence: 99%

ProFormer: Learning Data-efficient Representations of Body Movement with Prototype-based Feature Augmentation and Visual Transformers

Peng¹,

Roitberg²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…View Invariant Action Recognition. With the advancements in the field of Graph Convolutional Networks [18] and the availability of abundant 3D Pose data [42], many works have studied skeleton based action recognition [6,23,30,44,45,49,59]. These skeleton based methods are robust to viewpoint changes due to their extension across depth dimension.…”

Section: Related Workmentioning

confidence: 99%

ViewCLR: Learning Self-supervised Video Representation for Unseen Viewpoints

Das¹,

Ryoo²

2021

Preprint

View full text Add to dashboard Cite

Learning self-supervised video representation predominantly focuses on discriminating instances generated from simple data augmentation schemes. However, the learned representation often fails to generalize over unseen camera viewpoints. To this end, we propose ViewCLR, that learns self-supervised video representation invariant to camera viewpoint changes. We introduce a view-generator that can be considered as a learnable augmentation for any selfsupervised pre-text tasks, to generate latent viewpoint representation of a video. ViewCLR maximizes the similarities between the latent viewpoint representation with its representation from the original viewpoint, enabling the learned video encoder to generalize over unseen camera viewpoints. Experiments on cross-view benchmark datasets including NTU RGB+D dataset show that ViewCLR stands as a stateof-the-art viewpoint invariant self-supervised method.

show abstract

“…Thanks to the development of graph convolutional networks to process structured data, GCN-based methods [4], [51], [5], [6], [52], [7], [8], [9], [10], [11], [12], [53] have been demonstrated to be effective for skeleton-based action recognition. The latest GCN-based methods can be divided into three categories.…”

Section: B Skeleton-based Action Recognitionmentioning

confidence: 99%

“…The latest GCN-based methods can be divided into three categories. First, some methods design sophisticated GCN structures to better model the spatial and temporal features [10], [8], [53]. Liu et al [10] leverage dense cross space-time edges as skip connections for unbiased longrange and unobstructed information propagation in spatialtemporal dimension.…”

Section: B Skeleton-based Action Recognitionmentioning

confidence: 99%

“…Zhang et al [8] propose a contextaware GCN to consider a context term for each vertex by integrating information of all other vertices. Chen et al [53] propose a channel-wise topology refinement graph convolution to learn different topologies and aggregate joint features in different channels. Second, some models aim at reducing computational cost for skeleton data [7], [9], [12].…”

Section: B Skeleton-based Action Recognitionmentioning

confidence: 99%

See 1 more Smart Citation

Skeleton-Based Mutually Assisted Interacted Object Localization and Human Action Recognition

Li¹,

Lan²,

Zeng³

et al. 2021

Preprint

View full text Add to dashboard Cite

Skeleton data carries valuable motion information and is widely explored in human action recognition. However, not only the motion information but also the interaction with the environment provides discriminative cues to recognize the action of persons. In this paper, we propose a joint learning framework for mutually assisted "interacted object localization" and "human action recognition" based on skeleton data. The two tasks are serialized together and collaborate to promote each other, where preliminary action type derived from skeleton alone helps improve interacted object localization, which in turn provides valuable cues for the final human action recognition. Besides, we explore the temporal consistency of interacted object as constraint to better localize the interacted object with the absence of ground-truth labels. Extensive experiments on the datasets of SYSU-3D, NTU60 RGB+D and Northwestern-UCLA show that our method achieves the best or competitive performance with the state-of-the-art methods for human action recognition. Visualization results show that our method can also provide reasonable interacted object localization results.

show abstract

Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition

Cited by 340 publications

References 23 publications

ProFormer: Learning Data-efficient Representations of Body Movement with Prototype-based Feature Augmentation and Visual Transformers

ProFormer: Learning Data-efficient Representations of Body Movement with Prototype-based Feature Augmentation and Visual Transformers

ViewCLR: Learning Self-supervised Video Representation for Unseen Viewpoints

Skeleton-Based Mutually Assisted Interacted Object Localization and Human Action Recognition

Contact Info

Product

Resources

About