Spatio–Temporal Image Representation of 3D Skeletal Movements for View-Invariant Action Recognition with Deep Convolutional Neural Networks

Pham, Hieu H.; Salmane, Houssam; Khoudour, Louahdi; Crouzil, Alain; Zegers, Pablo; Velastín, Sergio A.

doi:10.3390/s19081932

Cited by 27 publications

(10 citation statements)

References 89 publications

(214 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hug et al, 2019 [35] applied an action recognition model based on the transformation of the skeleton to a spatial presentation using the conversion of the distance values of two joints to color points, and they used the DenseNet CNN model for action classification. We finish this section by presenting the major conclusions extracted by the two surveys of Wan et al, 2018 [36] and Pham et al, 2019 [37]. These authors found that methods based on human pose estimation and skeleton feature extraction can achieve higher classification rates.…”

Section: Related Workmentioning

confidence: 84%

Prediction of Human Activities Based on a New Structure of Skeleton Features and Deep Learning Model

Jaouedi¹,

Perales

Buades

et al. 2020

Sensors

View full text Add to dashboard Cite

The recognition of human activities is usually considered to be a simple procedure. Problems occur in complex scenes involving high speeds. Activity prediction using Artificial Intelligence (AI) by numerical analysis has attracted the attention of several researchers. Human activities are an important challenge in various fields. There are many great applications in this area, including smart homes, assistive robotics, human–computer interactions, and improvements in protection in several areas such as security, transport, education, and medicine through the control of falling or aiding in medication consumption for elderly people. The advanced enhancement and success of deep learning techniques in various computer vision applications encourage the use of these methods in video processing. The human presentation is an important challenge in the analysis of human behavior through activity. A person in a video sequence can be described by their motion, skeleton, and/or spatial characteristics. In this paper, we present a novel approach to human activity recognition from videos using the Recurrent Neural Network (RNN) for activity classification and the Convolutional Neural Network (CNN) with a new structure of the human skeleton to carry out feature presentation. The aims of this work are to improve the human presentation through the collection of different features and the exploitation of the new RNN structure for activities. The performance of the proposed approach is evaluated by the RGB-D sensor dataset CAD-60. The experimental results show the performance of the proposed approach through the average error rate obtained (4.5%).

show abstract

Section: Related Workmentioning

confidence: 84%

Prediction of Human Activities Based on a New Structure of Skeleton Features and Deep Learning Model

Jaouedi¹,

Perales

Buades

et al. 2020

Sensors

View full text Add to dashboard Cite

show abstract

“…The method allows to eliminate the coordinate deviations caused by various recording environments and posture displacements. Pham et al (2019) exploit Deep CNNs based on the DenseNet model to learn directly an end-toend mapping between the input skeleton sequences and their action label for human activity recognition. The network learns spatio-temporal patterns of skeletal movements and their discriminative features for further downstream classification tasks.…”

Section: Related Workmentioning

confidence: 99%

Detection of sitting posture using hierarchical image composition and deep learning

Kulikajevas

Maskeliūnas

Damaševičius

2021

PeerJ Computer Science

View full text Add to dashboard Cite

Human posture detection allows the capture of the kinematic parameters of the human body, which is important for many applications, such as assisted living, healthcare, physical exercising and rehabilitation. This task can greatly benefit from recent development in deep learning and computer vision. In this paper, we propose a novel deep recurrent hierarchical network (DRHN) model based on MobileNetV2 that allows for greater flexibility by reducing or eliminating posture detection problems related to a limited visibility human torso in the frame, i.e., the occlusion problem. The DRHN network accepts the RGB-Depth frame sequences and produces a representation of semantically related posture states. We achieved 91.47% accuracy at 10 fps rate for sitting posture recognition.

show abstract

“…Hinton et al [54] define a DNN as a neural network that consists of two or more hidden layers between the input layer and the output layer. The literature contains a number of techniques that employ deep-learning-based approaches; for example, [1,[55][56][57] rely on a Convolutional Neural Network (CNN), [19,[58][59][60] utilize a Recurrent Neural Network (RNN), as well as Long-Short Term Memory (LSTM), and [61] uses Deep Progress Reinforcement Learning (DPRL). Sedmidubsky et al [1] propose a method for action recognition and segmentation in which the motions are mapped onto encoded RGB images.…”

Section: Deep-learning-based Approachesmentioning

confidence: 99%

“…They extract the most informative frames from the input action video sequences through DPRL and employ a graph-based CNN in order to exploit the extrinsic, as well as intrinsic, human joint dependencies. Pham et al [57] propose an SPMF (Skeleton Posture-Motion Feature) based on necessary spatiotemporal information extracted from skeleton poses and their motions in order to represent unique patterns that exist in skeletal movements. It is further enhanced by exploiting the Adaptive Histogram Equalization (AHE) method to build the action map.…”

Section: Deep-learning-based Approachesmentioning

confidence: 99%

Keys for Action: An Efficient Keyframe-Based Approach for 3D Action Recognition Using a Deep Neural Network

Yasin

Hussain

Weber

2020

Sensors

View full text Add to dashboard Cite

In this paper, we propose a novel and efficient framework for 3D action recognition using a deep learning architecture. First, we develop a 3D normalized pose space that consists of only 3D normalized poses, which are generated by discarding translation and orientation information. From these poses, we extract joint features and employ them further in a Deep Neural Network (DNN) in order to learn the action model. The architecture of our DNN consists of two hidden layers with the sigmoid activation function and an output layer with the softmax function. Furthermore, we propose a keyframe extraction methodology through which, from a motion sequence of 3D frames, we efficiently extract the keyframes that contribute substantially to the performance of the action. In this way, we eliminate redundant frames and reduce the length of the motion. More precisely, we ultimately summarize the motion sequence, while preserving the original motion semantics. We only consider the remaining essential informative frames in the process of action recognition, and the proposed pipeline is sufficiently fast and robust as a result. Finally, we evaluate our proposed framework intensively on publicly available benchmark Motion Capture (MoCap) datasets, namely HDM05 and CMU. From our experiments, we reveal that our proposed scheme significantly outperforms other state-of-the-art approaches.2 of 24 inertial sensor-and accelerometer-based systems [7,8]; and hybrid systems [9]. As another domain, 3D motions are reconstructed from different sources, e.g., reconstruction from video or image data [10][11][12][13][14] and reconstruction from accelerometer data [15,16]. In short, motions captured or generated by different sources of means are abundant and contain a lot of hidden knowledge and information that may be exploited further in different types of applications, such as those mentioned above.There exists a variety of action classification methods that are based on different input data; the input may be in the form of simple RGB videos or spatiotemporal joint trajectories acquired by means of a sensor system (e.g., mechanical, magnetic, optic, inertial, non-optic wearable sensors and RGB-D sensors, such as Kinect) or an estimated 3D skeleton from image data or a hybrid system. Although a lot of research has been done in the domain of action recognition, there still exist numerous challenges, such as viewpoint variations, different human body sizes and appearances, and illumination factors that may influence the efficiency and performance of existing algorithms [17]. Moreover, each performing actor has his or her own way and style of executing the same action. The actions may also have many variations in terms of speed and length. In addition to these inter-class variations, extensive intra-class variations make the task more difficult. For example, it is not an easy task to differentiate between jogging and running, walkForward and walkBackward, sitDownChair and sitDownFloor, standUpSitChair and standUpSitFloor, rotateArmBackward and rota...

show abstract

Spatio–Temporal Image Representation of 3D Skeletal Movements for View-Invariant Action Recognition with Deep Convolutional Neural Networks

Cited by 27 publications

References 89 publications

Prediction of Human Activities Based on a New Structure of Skeleton Features and Deep Learning Model

Prediction of Human Activities Based on a New Structure of Skeleton Features and Deep Learning Model

Detection of sitting posture using hierarchical image composition and deep learning

Keys for Action: An Efficient Keyframe-Based Approach for 3D Action Recognition Using a Deep Neural Network

Contact Info

Product

Resources

About