Adaptive Feature Selection With Reinforcement Learning for Skeleton-Based Action Recognition

Abstract

Skeleton-based action recognition has attracted extensive attention recently in the computer vision community. Previous studies, especially GCN-based methods, have presented remarkable improvements for this task. However, in existing GCN-based methods, global average pooling is applied to the extracted features before the classifier. This may hurt the recognition performance since it neglects the fact that not all features are equally important in the temporal dimension. To tackle this issue, in this article, we propose a feature selection network (FSN) with actor-critic reinforcement learning. Given the extracted feature sequence, FSN learns to adaptively select the most representative features and discard the ambiguous features for action recognition. In addition, conventional graph convolution is a local operation, it cannot fully capture the non-local joint dependencies that could be vital to recognize the action. Thus, we also propose a generalized graph generation module to capture latent dependencies and further propose a generalized graph convolution network (GGCN). The GGCN and FSN are combined in a three-stream recognition framework, in which different types of information from skeleton data are further fused to improve the recognition accuracy. Extensive experiments demonstrate that the proposed FSN is a flexible and effective module that can cooperate with any existing GCN-based framework to enhance the recognition accuracy, the proposed GGCN can extract richer skeleton features for skeleton-based action recognition, and our method achieves superior performance over several public datasets, e.g. 95.7 top-1 accuracy on NTU-RGB+D, 86.7 top-1 accuracy on NTU-RGB+D 120, etc.

Skeleton Graph-Neural-Network-Based Human Action Recognition: A Survey

“…R. Zhao et al [ 69 ] performed GCN and LSTM separately, the spatial information from GCN in each frame was directly input into LSTM cell. Z. Y. Xu et al [ 91 ] proposed using RL combined with LSTM as the feature selection network (FSN) consisting of a policy network and a value network. To be precise, both the policy network and value network are based on LSTM for sequential action or value generation.…”

“…The first benefit has been proven by numerous papers using CNN or RNN. Methods, such as [ 51 , 61 , 67 , 91 , 107 , 110 , 113 , 120 , 121 , 122 , 123 , 126 , 128 , 132 , 133 , 134 , 135 ], all follow the basic architecture of skip connection. The second benefit was also discovered by multiple papers.…”

“…Refs. [ 54 , 91 ] failed while classifying reading, writing, playing with phone/tablet, and typing on a keyboard . The authors argue these actions only differ for hand movements, while the skeletons provided by NTU RGB+D are less supportive for hand joints.…”

“…One is object detection, the other is HAR according to the semantics of object detection. Human–objects interactions that have subtle movements of hands can be easily misidentified [ 83 , 91 , 92 ]. However, Ref.…”

Deep Reinforcement Learning in Human Activity Recognition: A Survey

“…For skeleton-based activity recognition, [59] proposed a feature selection network (FSN) with Actor-Critic RL algorithm. This is to select the most descriptive frames and discard ambiguous frames in a sequence.…”