Learning Multi-Granular Spatio-Temporal Graph Network for Skeleton-based Action Recognition

Chen, Tailin; Zhou, Desen; Wang, Jian; Wang, Shidong; Guan, Yu; He, Xuming; Ding, Errui

doi:10.48550/arxiv.2108.04536

Cited by 1 publication

(1 citation statement)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Thirty-Sixth AAAI Conference on Artificial Intelligence property / method (group) Fan et al 2020;Yang et al 2020b;Li et al 2019b;Qin et al 2021;Chen et al 2021a;Plizzari, Cannici, and Matteucci 2020;Shi et al 2020a,b;Lei et al 2019), G 3 : (Li, Zhang, and Li 2020;Pan, Chen, and Ortega 2021;Cho et al 2020;Peng et al 2020;Si et al 2018;Song et al 2017a;Xie et al 2018;Huang et al 2020;Si et al 2019).…”

Section: (C) Top Row) the Static Topological Connectionsmentioning

confidence: 99%

Towards To-a-T Spatio-Temporal Focus for Skeleton-Based Action Recognition

Ke¹,

Peng²,

Lyu³

2022

AAAI

View full text Add to dashboard Cite

Graph Convolutional Networks (GCNs) have been widely used to model the high-order dynamic dependencies for skeleton-based action recognition. Most existing approaches do not explicitly embed the high-order spatio-temporal importance to joints’ spatial connection topology and intensity, and they do not have direct objectives on their attention module to jointly learn when and where to focus on in the action sequence. To address these problems, we propose the To-a-T Spatio-Temporal Focus (STF), a skeleton-based action recognition framework that utilizes the spatio-temporal gradient to focus on relevant spatio-temporal features. We first propose the STF modules with learnable gradient-enforced and instance-dependent adjacency matrices to model the high-order spatio-temporal dynamics. Second, we propose three loss terms defined on the gradient-based spatio-temporal focus to explicitly guide the classifier when and where to look at, distinguish confusing classes, and optimize the stacked STF modules. STF outperforms the state-of-the-art methods on the NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400 datasets in all 15 settings over different views, subjects, setups, and input modalities, and STF also shows better accuracy on scarce data and dataset shifting settings.

show abstract