Enhanced Spatial and Extended Temporal Graph Convolutional Network for Skeleton-Based Action Recognition

Abstract

In the skeleton-based human action recognition domain, the spatial-temporal graph convolution networks (ST-GCNs) have made great progress recently. However, they use only one fixed temporal convolution kernel, which is not enough to extract the temporal cues comprehensively. Moreover, simply connecting the spatial graph convolution layer (GCL) and the temporal GCL in series is not the optimal solution. To this end, we propose a novel enhanced spatial and extended temporal graph convolutional network (EE-GCN) in this paper. Three convolution kernels with different sizes are chosen to extract the discriminative temporal features from shorter to longer terms. The corresponding GCLs are then concatenated by a powerful yet efficient one-shot aggregation (OSA) + effective squeeze-excitation (eSE) structure. The OSA module aggregates the features from each layer once to the output, and the eSE module explores the interdependency between the channels of the output. Besides, we propose a new connection paradigm to enhance the spatial features, which expand the serial connection to a combination of serial and parallel connections by adding a spatial GCL in parallel with the temporal GCLs. The proposed method is evaluated on three large scale datasets, and the experimental results show that the performance of our method exceeds previous state-of-the-art methods.

Whole and Part Adaptive Fusion Graph Convolutional Networks for Skeleton-Based Action Recognition

“…Bone has proven to be another form of spatial information as important as joints and has been used in many recent methods [ 32 , 37 , 38 , 41 , 42 , 43 , 44 ]. The original skeleton data only contain the 3D coordinates of all joints in the skeleton, and the bone stream data are obtained by vector calculation of the original joint stream data.…”

Skeleton Graph-Neural-Network-Based Human Action Recognition: A Survey

“…Methods, such as [ 49 , 73 , 74 , 100 , 120 , 126 , 127 , 131 ], assumed that features in different channels have various importance, and thus they attempted to balance the importance of each channel while inferring, known as channel-wise attention.…”

“…There are mainly two types of dense block structures. One is the SE block [ 120 ], working on channel dimension, which performs a channel-wise attention for adaptive aggregation along the channel dimension. The other one is skip connection, also known as residual connection, which adds more connections between hidden states.…”

“…The first benefit has been proven by numerous papers using CNN or RNN. Methods, such as [ 51 , 61 , 67 , 91 , 107 , 110 , 113 , 120 , 121 , 122 , 123 , 126 , 128 , 132 , 133 , 134 , 135 ], all follow the basic architecture of skip connection. The second benefit was also discovered by multiple papers.…”

“…Methods with SE block highlight the non-homogeneous features in each channel and attempt to squeeze features by pooling. One example of effective SE (eSE) is demonstrated in Figure 10 c. F. Li et al [ 120 ] concatenate consecutive layers by a one-shot aggregation (OSA) and the effective squeeze-excitation (eSE) block, an improved version of SE, which well balances the performance and efficiency. The eSE explores the interdependency between the output channels and squeezes the temporal and spatial dependencies.…”