Feedback Graph Convolutional Network for Skeleton-Based Action Recognition

Abstract

Skeleton-based action recognition has attracted considerable attention since the skeleton data is more robust to the dynamic circumstances and complicated backgrounds than other modalities. Recently, many researchers have used the Graph Convolutional Network (GCN) to model spatial-temporal features of skeleton sequences by an end-to-end optimization. However, conventional GCNs are feedforward networks for which it is impossible for the shallower layers to access semantic information in the high-level layers. In this paper, we propose a novel network, named Feedback Graph Convolutional Network (FGCN). This is the first work that introduces a feedback mechanism into GCNs for action recognition. Compared with conventional GCNs, FGCN has the following advantages:(1) A multi-stage temporal sampling strategy is designed to extract spatial-temporal features for action recognition in a coarse to fine process; (2) A Feedback Graph Convolutional Block (FGCB) is proposed to introduce dense feedback connections into the GCNs. It transmits the high-level semantic features to the shallower layers and conveys temporal information stage by stage to model video level spatial-temporal features for action recognition; (3) The FGCN model provides predictions on-the-fly. In the early stages, its predictions are relatively coarse. These coarse predictions are treated as priors to guide the feature learning in later stages, to obtain more accurate predictions. Extensive experiments on three datasets, NTU-RGB+D, NTU-RGB+D120 and Northwestern-UCLA, demonstrate that the proposed FGCN is effective for action recognition. It achieves the state-of-the-art performance on all three datasets.

Skeleton Graph-Neural-Network-Based Human Action Recognition: A Survey

“…Multi-stream is the most commonly used structure in skeleton-GNN-HAR, methods, such as [ 14 , 49 , 52 , 54 , 68 , 69 , 71 , 73 , 87 , 89 , 90 , 93 , 99 , 104 , 114 , 119 , 120 , 123 , 125 , 126 , 134 , 137 , 138 , 139 ] all use this framework. This framework utilizes different types of data, such as joint stream, bone stream, part stream, relative coordinates of the joints, temporal displacements.…”

“…The colors identity each dataset. In ( b ), the numbers around dots denote [ 12 , 13 , 29 , 56 , 59 , 61 , 69 , 72 , 74 , 75 , 83 , 85 , 87 , 88 , 89 , 90 , 92 , 97 , 110 , 115 , 135 , 136 , 138 , 141 , 182 ] respectively in ascending order.…”

“… CAD 120 [ 67 ] Make Cereal, Take Medicine, Stack Objects, Unstack Objects, Microwave Food, Pick Objects, Clean Objects, Take Food, Arrange Objects, Have a Meal ORGBD [ 115 ] Drink, Eat, Use Laptop, Read Cellphone, Make Phone Call, Read Book, Use Remote Human3.6M [ 95 ] Conversations, Eat, Greet, Talk on the Phone, Pose, Sit, Smoke, Take Photos, Wait, Walk in Various Non-Typical Scenarios (With a Hand in the Pocket, Talk on the Phone, Walk a Dog, or Buy an Item) N-UCLA [ 59 , 87 , 89 , 90 , 92 , 94 , 138 , 141 ] Pick Up With One Hand, Pick Up With Two Hands, Drop Trash, Walk Around, Sit Down, Stand Up, Donning, Doffing, Throw, Carry UWA3D Multiview [ 29 ] One Hand Wave, One Hand Punch, Sit Down, Stand Up, Hold Chest, Hold Head, Hold Back, Walk, Turn Around, Drink, Bend, Run, Kick, Jump, Mope Floor, Sneeze, Sit Down (Chair), Squat, Two Hand Wave, Two Hand Punch, Vibrate, Fall Down, Irregular Walk, Lie Down, Phone Answer, Jump Jack, Pick Up, Put Down, Dance, Cough UWA3D Multiview Activity II [ 29 , 94 ] One Hand Wave, One Hand Punch, Two Hand Wave, Two Hand Punch, Sit Down, Stand Up, Vibrate, Fall Down, Hold Chest, Hold Head, Hold Back, Walk, Irregular Walk, Lie Down, Turn Around, Drink, Phone Answer, Bend, Jump Jack, Run, Pick Up, Put Down, Kick, Jump, Dance, Mope Floor, Sneeze, Sit Down (Chair), Squat, Cough UTD-MHAD [ 69 , 105 ] …”

Online Skeleton-based Action Recognition with Continual Spatio-Temporal Graph Convolutional Networks

“…Hyper-GNN [19] captured the non-physical connections between the nodes by constructing hyperedges which help to extract both local and global features in each graph. FGCN [20] proposed to extract coarse to fine spatio-temporal features by a multistage temporal sampling strategy and introduced a feedback mechanism in graph convolution to transfer the high-level features to the shallower layers of the network. Similarly, MS-G3D [21] has proposed multi-scale graph convolutions for long-range feature extraction.…”

Deep learning‐based action recognition with 3D skeleton: A survey

“…The detailed architecture of the proposed FGCB local network [59]. F I G U R E 1 2 (a) It shows the original spatial and temporal modeling on skeleton graph sequences of GCN-based methods.…”