A Deep Learning Approach for Real-Time 3D Human Action Recognition from Skeletal Data

Abstract

We present a new deep learning approach for real-time 3D human action recognition from skeletal data and apply it to develop a vision-based intelligent surveillance system. Given a skeleton sequence, we propose to encode skeleton poses and their motions into a single RGB image. An Adaptive Histogram Equalization (AHE) algorithm is then applied on the color images to enhance their local patterns and generate more discriminative features. For learning and classification tasks, we design Deep Neural Networks based on the Densely Connected Convolutional Architecture (DenseNet) to extract features from enhanced-color images and classify them into classes. Experimental results on two challenging datasets show that the proposed method reaches state-of-the-art accuracy, whilst requiring low computational time for training and inference. This paper also introduces CEMEST, a new RGB-D dataset depicting passenger behaviors in public transport. It consists of 203 untrimmed real-world surveillance videos of realistic normal and anomalous events. We achieve promising results on real conditions of this dataset with the support of data augmentation and transfer learning techniques. This enables the construction of real-world applications based on deep learning for enhancing monitoring and security in public transport.

3D human pose data augmentation using Generative Adversarial Networks for robotic-assisted movement quality assessment

“…This research focuses on using deep learning techniques to enhance 3D human motion data. Specifically, it explores how Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) can be utilized to process and generate more diverse and rich human motion data (Pham et al, 2019). These techniques can learn from existing small datasets and generate new data samples to support more complex motion analysis and machine learning applications (Wang et al, 2018).…”

Video-based Human Action Recognition using Deep Learning: A Review

“…Researchers have successfully applied CNN-based architectures for many visual tasks such as people detection and tracking [126], [127], [128], pose estimation [129], [130], [131], [132], [133], [134], action recognition [79], [135], [136], [137], [138], [139], [140], [141], [142], [143], [144], [145], [146], [147], [148], [149], [150], [151], [152], [153], [154], [155], [156], [157], [158], event detection and crowded scene understanding [159], [160], [161], [162]. Early work on applying CNNs was made in 1995 by Nowlan et al [129] for hand tracking and recognizing.…”

Recognition of Rare Low-Moral Actions Using Depth Data

“…A number of works employed recurrent neural networks (RNN) or long-short-term memory (LSTM) structures to model the spatial-temporal evolution of the skeleton joints [6,[23][24][25]. A different approach is to treat the skeleton data as a pseudo-image, which turns the action recognition into a image based classification problem, for which convolutional neural networks (CNNs) can be used [26][27][28][29].…”

“…Very few works used transfer learning to improve the action recognition on depth or skeleton data. Pham et al [29] used pretraining on large action datasets to improve the performance of skeleton based action recognition on their smaller dataset of passenger behavior in a metro station (the dataset included also low-moral actions like jumping over or sneaking under ticket barriers). In this work we used transfer learning in a similar way, and tested it for both depth and skeleton data based action recognition.…”

“…That is particularly true for depth data since not many datasets for that purpose are publicly available. One example of research on low-moral action recognition is the just mentioned metro station study in [29].…”