Action-Attending Graphic Neural Network

Abstract

The motion analysis of human skeletons is crucial for human action recognition, which is one of the most active topics in computer vision. In this paper, we propose a fully end-to-end action-attending graphic neural network (A2GNN) for skeleton-based action recognition, in which each irregular skeleton is structured as an undirected attribute graph. To extract high-level semantic representation from skeletons, we perform the local spectral graph filtering on the constructed attribute graphs like the standard image convolution operation. Considering not all joints are informative for action analysis, we design an actionattending layer to detect those salient action units (AUs) by adaptively weighting skeletal joints. Herein the filtering responses are parameterized into a weighting function irrelevant to the order of input nodes. To further encode continuous motion variations, the deep features learnt from skeletal graphs are gathered along consecutive temporal slices and then fed into a recurrent gated network. Finally, the spectral graph filtering, action-attending and recurrent temporal encoding are integrated together to jointly train for the sake of robust action recognition as well as the intelligibility of human actions. To evaluate our A2GNN, we conduct extensive experiments on four benchmark skeletonbased action datasets, including the large-scale challenging NTU RGB+D dataset. The experimental results demonstrate that our network achieves the state-of-the-art performances.

Action Recognition With Spatio–Temporal Visual Attention on Skeleton Image Sequences

“…Other effective approaches include lie groups [10], [33] and nearest neighbor search [34]. Recently, graphical neural networks [35], [36] achieve the state-of-the-art performance on the skeleton based recognition.…”

Aggressive Action Estimation: A Comprehensive Review on Neural Network Based Human Segmentation and Action Recognition

“…Jé gou et al 2017 Though they provide good result, their usage of pooling technique limits the ability of fully utilizing input data. In Table 1 [19], authors proposed a new graph based neural network with a fully connected layer for action analysis, where action attending layer produces salient action units. While in Edwards et al (2016) [29], graph theorem on irregular domain was used in conjunction with convolution and pooling for image processing.…”

“…Rahmani et al (2018) [15], [16], [17],Li et al (2018) [18] andLi et al (2018) [19] used skeleton information to train their system Rahmani et al (2018) [15]. used dense trajectory calculation and encoded the shape of the trajectory for training and learned through knowledge transfer model while Zhang et al(2017) [16] and Zhang et al (2017) [17] utilized LSTM to extract and learn temporal dynamics.…”

Spatio-Temporal Graph Routing for Skeleton-Based Action Recognition

“…In fact, in the research fields of graph classification/matching, the graph representation (e.g., the statistic on graphlet (Pržulj 2007)) and graph metric (e.g., graph kernel (Vishwanathan et al 2010)) have been historically well-studied. With recent successes of deep learning on various problems, deep representation of graphs has aroused more attention (Yanardag and Vishwanathan 2015;Seo et al 2016;Li et al 2017). But the most crucial problem is the definition/identification of homogeneous graphs because the same responses should be produced from those homogeneous graphs.…”

“…Jain et al (Jain et al 2016) proposed a structural-RNN by casting spatio-temporal graph as a RNN mixture for the task of action prediction. Li et al (Li et al 2015) proposed gated graph sequential neural network for the basic logical reasoning task. Seo et al (Seo et al 2016) fed the spatial filtered graph signals into LSTM for image generation.…”