Deep Convolutional Neural Networks for Human Action Recognition Using Depth Maps and Postures

Abstract

In this paper, we present a method for human action recognition from depth images and posture data using convolutional neural networks (CNN). Two input descriptors are used for action representation. The first input is a depth motion image (DMI) that accumulates consecutive depth images of a human action, whilst the second input is a proposed moving joints descriptor (MJD) which represents the motion of body joints over time. In order to maximize feature extraction for accurate action classification, three CNN channels are trained with different inputs. The first channel is trained with depth motion images, the second channel is trained with both depth motion images and moving joint descriptors together, and the third channel is trained with moving joint descriptors only. The action predictions from the three CNN channels are fused together for the final action classification. We propose several fusion score operations to maximize the score of the right action. The experiments show that the results of fusing the output of three channels are better than using one channel or fusing two channels only. Our proposed method was evaluated on three public datasets: MSRAction3D, UTD-MAHD, and MAD dataset. The testing results indicate that the proposed approach outperforms most of existing state of the art methods such as HON4D and Actionlet on MSRAction3D. Although MAD dataset contains a high number of actions (35 actions) compared to existing action RGB-D datasets, our work achieves 91.86% of accuracy.

Efficient Body Motion Quantification and Similarity Evaluation Using 3-D Joints Skeleton Coordinates

“…In Fig. 6(a) top, we fix the angle thresholds at THAngles = [12,12,10,10,8,5,8,5] and we changed the THiRT thresholds from TH1RT to TH7RT, then we check the comparison results. We notice that from TH4RT to TH7RT, the algorithm generates correct judgments for the four actions pairs.…”

Multimodal representation models for prediction and control from partial information

“…Imitation can happen at different levels, such as at the action level, or at the effect level [41]. Recently, advances on motion analysis and estimation have been proposed [13,14,15], and these techniques have also been applied to humanoid robot motion learning through sensorimotor representation and physical interactions [42]. In this paper, we use a trajectory level imitation, as an instrumental example of application of our proposed multimodal learning approach.…”

“…learning a forward model) or actions of others (e.g. human trajectories from images or videos) [13,14,15], in this paper the goal is to learn a model of the self that can be applied to predict and imitate the visual perception of another agent from an egocentric point of view. The proposed architecture is based on a self-learned model, which is built, trained and updated only using the experience accumulated by the agent.…”

A Review on Human Action Recognition in Surveillance Videos

“…This represents how the body joints are moving during action. [2] There are three Convolution Neural Networks of DMI outputs only, both DMI and MJD outputs and the third containing MJD outputs only. These results are merged to get the final result.…”

“…In 1991 it was modified and was used for medical image processing and automatic detection of breast cancer in mammograms. Another convolution-based design was proposed in 1988 which was applied to decompose 1D electromyography convolved signals via deconvolution [2]. Another met`hod for this is studying skeleton joints called temporal pyramid or RGB-D dataset.…”