Topology-Aware Convolutional Neural Network for Efficient Skeleton-Based Action Recognition

Abstract

In the context of skeleton-based action recognition, graph convolutional networks (GCNs) have been rapidly developed, whereas convolutional neural networks (CNNs) have received less attention. One reason is that CNNs are considered poor in modeling the irregular skeleton topology. To alleviate this limitation, we propose a pure CNN architecture named Topology-aware CNN (Ta-CNN) in this paper. In particular, we develop a novel cross-channel feature augmentation module, which is a combo of map-attend-group-map operations. By applying the module to the coordinate level and the joint level subsequently, the topology feature is effectively enhanced. Notably, we theoretically prove that graph convolution is a special case of normal convolution when the joint dimension is treated as channels. This confirms that the topology modeling power of GCNs can also be implemented by using a CNN. Moreover, we creatively design a SkeletonMix strategy which mixes two persons in a unique manner and further boosts the performance. Extensive experiments are conducted on four widely used datasets, i.e. N-UCLA, SBU, NTU RGB+D and NTU RGB+D 120 to verify the effectiveness of Ta-CNN. We surpass existing CNN-based methods significantly. Compared with leading GCN-based methods, we achieve comparable performance with much less complexity in terms of the required GFLOPs and parameters.

Motion-Driven Spatial and Temporal Adaptive High-Resolution Graph Convolutional Networks for Skeleton-Based Action Recognition

“…Year CS(%) CV(%) GCA-LSTM [17] 2018 74.3 82.8 ST-GCN [25] 2018 81.5 88.3 SR-TSL [15] 2018 84.8 92.4 HCN [69] 2018 86.5 91.1 DPRL+GCNN [35] 2018 83.5 89.8 AS-GCN [50] 2019 86.8 94.2 AGC-LSTM [70] 2019 89.2 95.0 2s-AGCN [26] 2019 88.5 95.1 DGNN [39] 2019 89.9 96.1 BAGCN [61] 2019 90.3 96.3 STGR-GCN [60] 2019 86.9 92.3 TS-SAN [71] 2020 87.2 92.7 2s-AAGCN+TEM [47] 2020 88.7 95.8 SGN [72] 2020 89.0 94.5 2s-Shift-GCN [29] 2020 89.7 96 GCN-NAS [48] 2020 89.4 95.7 2s-AAGCN [27] 2020 89.4 96.0 MS-AAGCN [27] 2020 90.0 96.2 CGCN [63] 2020 90.3 96.4 4s-Shift-GCN [29] 2020 90.7 96.5 MS-AAGCN+TEM [47] 2020 91 96.5 Dynamic GCN [30] 2020 91.5 96 MS-G3D [28] 2020 91.5 96.2 PR-GCN [64] 2021 85.2 91.7 RA-GCN [49] 2021 87.3 93.6 SEFN(Base) [31] 2021 89.2 95.8 ST-TR [36] 2021 89.9 96.1 SEFN(Att) [31] 2021 90.2 96.1 ST-TR-agcn [36] 2021 90.3 96.3 MSTGNN [73] 2021 91.3 95.5 PB-GCN [65] 2022 83.8 91.3 PeGCN [66] 2022 85.6 93.4 LAGA [51] 2022 87.1 93.2 TE-GCN [68] 2022 88.7 95.4 EGCN [67] 2022 89.1 95.5 Graph2Net [32] 2022 90.1 96 Sybio-GNN [33] 2022 90.1 95.4 CD-GCN [34] 2022 90.1 96.5 Ta-CNN [24] 2022 90.7 95.1 FR-AGCN [52] 2022 90. is c...…”

“…Year CS(%) CV(%) GCA-LSTM [17] 2018 61.2 63.3 RotClips+MTCNN [74] 2018 62.2 61.8 BPEM [75] 2018 64.6 66.9 ST-GCN [25] 2018 70.7 73.2 SR-TSL [15] 2018 74.1 79.9 TSRJI [76] 2019 67.9 62.8 2s-AGCN [26] 2019 82.9 84.9 SGN [72] 2020 79.2 81.5 2s-Shift-GCN [29] 2020 85.3 86.6 4s-Shift-GCN [29] 2020 85.9 87.6 MS-G3D [28] 2020 86.9 88.4 Dynamic GCN [30] 2020 87.3 88.6 RA-GCN [49] 2021 81.1 82.7 ST-TR [36] 2021 84.3 86.7 ST-TR-agcn [36] 2021 85.1 87.1 SEFN [31] 2021 86.2 87.8 MSTGNN [73] 2021 87.4 87.6 LAGA [51] 2022 81 82.2 Ta-CNN [24] 2022 85.7 87.3 Graph2Net [32] 2022 86 87.6 CD-GCN [34] 2022 86.3 87.8 FR-AGCN [52] 2022 86.…”

“…Dhiman et al [23] projected a skeleton sequence onto an image and encrypted the human poses using different colors.The pseudo image is then classified using a part-wise spatio-temporal attentiondriven CNN network. Xu et al [24] proposed a pure CNN architecture named topology-aware CNN (Ta-CNN) with a cross-channel feature augmentation module. Although a lot of efforts have been made in CNN-based methods, it is still a challenge to model spatio-temporal information for action recognition.…”

“…Early skeleton-based methods focus on extracting handcrafted features [11]- [13], or encode the skeleton data into sequential vectors or pseudo images, and then model and classify them with recurrent neural networks (RNN) [14], [15], long short-term memory (LSTM) networks [16]- [18], or convolutional neural networks (CNN) [19]- [24]. However, these methods either break the natural spatial graph structure of skeleton data or make it difficult to extract temporal features, and thus cannot fully model the complex spatiotemporal configurations and correlations of the body joints for HAR [1].…”

Fully Attentional Network for Skeleton-Based Action Recognition

“…However, these methods are mostly combined with graph convolutional networks, and are thus constrained by the fixed graph structure. Later methods mainly make efforts in enlarging the receptive field [28], [30], [34], [35], [39], [40], [41], combining another stream [28], [30], [32], [35], [42], dividing graphs into structural ones [30], [38] and combining adaptive learning [28], [35], [43], [44].…”