Online Skeleton-based Action Recognition with Continual Spatio-Temporal Graph Convolutional Networks

Abstract

Graph-based reasoning over skeleton data has emerged as a promising approach for human action recognition. However, the application of prior graph-based methods, which predominantly employ whole temporal sequences as their input, to the setting of online inference entails considerable computational redundancy. In this paper, we tackle this issue by reformulating the Spatio-Temporal Graph Convolutional Neural Network as a Continual Inference Network, which can perform step-by-step predictions in time without repeat frame processing. To evaluate our method, we create a continual version of ST-GCN, CoST-GCN, alongside two derived methods with different self-attention mechanisms, CoAGCN and CoS-TR. We investigate weight transfer strategies and architectural modifications for inference acceleration, and perform experiments on the NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400 datasets. Retaining similar predictive accuracy, we observe up to 109× reduction in time complexity, on-hardware accelerations of 26×, and reductions in maximum allocated memory of 52% during online inference.

Continual Inference: A Library for Efficient Online Inference with Deep Neural Networks in PyTorch

“…Through a reformulation of the 3D convolution to compute inputs step-by-step rather than spatio-temporally, well-performing 3D CNNs such as X3D (Feichtenhofer, 2020), Slow (Feichtenhofer et al, 2019, and I3D (Carreira & Zisserman, 2017) trained for Trimmed Activity Recognition were re-implemented to execute step-by-step without any re-training. Likewise, Spatio-temporal Graph Convolutional Networks for Skeleton-based Action Recognition (Yan et al, 2018;Shi et al, 2019;Plizzari et al, 2021), which originally operated only on batches, were recently transformed to perform step-wise inference as well though a continual formulation of their Spatiotemporal Graph Convolution blocks (Hedegaard et al, 2022b).…”

“…In general, non-continual networks, which are transformed to continual ones attain reductions in per-step computational complexity in proportion to the temporal receptive field of the network. In some cases, these savings can amount to multiple orders of magnitude (Hedegaard et al, 2022b). Still, the implementation of Continual Inference Networks with temporal convolutions and Multi-head Attention in frameworks such as PyTorch (Paszke et al, 2019) requires deep knowledge and practical experience with CINs.…”

“…accuracy was improved by increasing model receptive fields through expansions of temporal global average pooling to 64 steps, and in Hedegaard et al (2022b), the stride of temporal convolutions was reduced to one to increase prediction rates. Sec.…”

“…The noted metrics were originally presented inHedegaard & Iosifidis (2021);Hedegaard et al (2022a,b). and CoS-TR for Skeleton-based Action Recognition inHedegaard et al (2022b). While direct conversion from regular to continual versions of the above noted architectures works well in accelerating inference in itself, further improvements can be achieved by exploiting some core characteristics of CINs: in Hedegaard & Iosifidis (2021),…”

Skeleton based human action recognition with relative position encoding

“…Heidari and Iosifidis 20 proposed the TA-GCN model to select the key bones most conducive to activity recognition to perform spatiotemporal convolution operations on the skeleton sequence. In order to realize online human behavior recognition, Hedegaard and Heidari et al 21 proposed Continual Spatio-Temporal Graph Convolutional Network (CoST-GCN), which reorganized the spatio-temporal graph convolutional neural network as a continuous inference network, which was processed without frame repetition. In this case, the step-by-step prediction function on the time axis is implemented.…”