Group-Skeleton-Based Human Action Recognition in Complex Events

Abstract

Human action recognition as an important application of computer vision has been studied for decades. Among various approaches, skeleton-based methods recently attract increasing attention due to their robust and superior performance. However, existing skeleton-based methods ignore the potential action relationships between different persons, while the action of a person is highly likely to be impacted by another person especially in complex events. In this paper, we propose a novel groupskeleton-based human action recognition method in complex events. This method first utilizes multi-scale spatial-temporal graph convolutional networks (MS-G3Ds) to extract skeleton features from multiple persons. In addition to the traditional key point coordinates, we also input the key point speed values to the networks for better performance. Then we use multilayer perceptrons (MLPs) to embed the distance values between the reference person and other persons into the extracted features. Lastly, all the features are fed into another MS-G3D for feature fusion and classification. For avoiding class imbalance problems, the networks are trained with a focal loss. The proposed algorithm is also our solution for the Large-scale Human-centric Video Analysis in Complex Events Challenge. Results on the HiEve dataset show that our method can give superior performance compared to other state-of-the-art methods.

Skeleton Graph-Neural-Network-Based Human Action Recognition: A Survey

“…Coping with this, papers [ 48 , 187 , 188 ] concentrate on relative features, paper [ 101 ] splits the motion model into single person and double person motion model, paper [ 115 ] attempted to transfer the knowledge between the two interacted skeletons by the teacher–student mechanism. Ref.…”

“…and 26 Mutual Actions (Handshaking, Pushing etc.). IRD [ 177 ] Garbage Dump, Normal HiEve [ 188 ] Walk-Alone, Walk-Together, Run-Alone, Run-Together, Ride, Sit-Talk, Sit-Alone, Queuing, Stand-Alone, Gather, Fight, Fall-Over, Walk-Up-Down-Stairs and Crouch-Bow Examples of each dataset are listed in Figure A1 , Figure A2 and Figure A3 .

Figure A1 Examples of datasets.

…”

“…+ temporal conv. 39.4 96.8 91.4 - - EN-GCN [ 193 ] - 2020 Fuse edge and node - 91.6 83.2 - - MS TE-GCN [ 194 ] - 2020 GCN+1DCNN as TCN - 96.2 90.8 - 84.4 ST-GCN+channel augmentation [ 103 ] - 2020 ST-GCN+new features from parameterized curve - 91.3 83.4 - 77.3 RHCN+ACSC + STUFE [ 182 ] - 2020 CNN, skeleton alignment - 92.5 86.9 - - [ 188 ] - 2020 Use MS-G3D to extract features, multiple person …”

Towards Building a Visual Behaviour Analysis Pipeline for Suicide Detection and Prevention

“…Kim et al [14] used skeleton-based action recognition to recognise the behaviour of persons handling objects such as a phone, a cup or a plastic bag in visual surveillance systems. Li et al [2] proposed a novel group-skeleton-based human action recognition method for complex events. However, research on CCTV video analysis for suicide detection and prevention is very limited.…”

“…Understanding of human behaviours through video analysis has boomed in recent years with the advances in deep learning and the ubiquity of cameras. There is growing interest in applying it to visual surveillance systems [1][2][3]. They have been widely deployed in both public and private locations, such as transport systems, parks, shopping centres, hospitals and prisons.…”