Action Recognition Based on Joint Trajectory Maps Using Convolutional Neural Networks

Wang, Pichao; Li, Zhaoyang; Hou, Yonghong; Li, Wanqing

doi:10.1145/2964284.2967191

Cited by 257 publications

(214 citation statements)

References 22 publications

Supporting

Mentioning

212

Contrasting

Order By: Relevance

“…To verify the differences from the other DTW algorithms, we compared DTW and modified DTW. In comparison with the general methods, we used discrete HMM, SVM (for MSRC-12 datasets only) [22], and two CNN methods (SOS, JTM) [23,24]. The 1-NN method was used to find the most similar movement in the gesturerecognition system, and the threshold-value setting of the gesture-recognition system was used to filter out insignificant gestures.…”

Section: Comparison and Analysis Of Dtw Performance Based On Dsmentioning

confidence: 99%

Modified Dynamic Time Warping Based on Direction Similarity for Fast Gesture Recognition

Choi

Kim

2018

Mathematical Problems in Engineering

View full text Add to dashboard Cite

We propose a modified dynamic time warping (DTW) algorithm that compares gesture-position sequences based on the direction of the gestural movement. Standard DTW does not specifically consider the two-dimensional characteristic of the user's movement. Therefore, in gesture recognition, the sequence comparison by standard DTW needs to be improved. The proposed gesturerecognition system compares the sequences of the input gesture's position with gesture positions saved in the database and selects the most similar gesture by filtering out unrelated gestures. The suggested algorithm uses the cosine similarity of the movement direction at each moment to calculate the difference and reflects the characteristics of the gesture movement by using the ratio of the Euclidean distance and the proportional distance to the calculated difference. Selective spline interpolation assists in solving the issue of recognition-decline at instances of gestures. Through experiments with public databases (MSRC-12 and G3D), the suggested algorithm revealed an improved performance on both databases compared to other methods.

show abstract

Section: Comparison and Analysis Of Dtw Performance Based On Dsmentioning

confidence: 99%

Modified Dynamic Time Warping Based on Direction Similarity for Fast Gesture Recognition

Choi

Kim

2018

Mathematical Problems in Engineering

View full text Add to dashboard Cite

show abstract

“…We compare it to other skeleton representations in the literature. Besides the classical skeleton image representation of Du et al [4], we compare with other representations used by state-ofthe-art approaches [32,10,13,14,34] as baselines on NTU RGB+D 60 [25]. We also compare our approach to sate-of-the-art methods on the NTU RGB+D 120 [15].…”

Section: Resultsmentioning

confidence: 99%

“…Wang et al [32,30] present a skeleton representation to represent both spatial configuration and dynamics of joint trajectories into three texture images through color encoding, named Joint Trajectory Maps (JTMs). The authors apply rotations to the skeleton data to mimicking multi-views and also for data enlargement to overcome the drawback of CNNs usually being not view invariant.…”

Section: Related Workmentioning

confidence: 99%

SkeleMotion: A New Representation of Skeleton Joint Sequences based on Motion Information for 3D Action Recognition

Caetano

Sena

Bremond

et al. 2019

2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)

156

View full text Add to dashboard Cite

Due to the availability of large-scale skeleton datasets, 3D human action recognition has recently called the attention of computer vision community. Many works have focused on encoding skeleton data as skeleton image representations based on spatial structure of the skeleton joints, in which the temporal dynamics of the sequence is encoded as variations in columns and the spatial structure of each frame is represented as rows of a matrix. To further improve such representations, we introduce a novel skeleton image representation to be used as input of Convolutional Neural Networks (CNNs), named SkeleMotion. The proposed approach encodes the temporal dynamics by explicitly computing the magnitude and orientation values of the skeleton joints. Different temporal scales are employed to compute motion values to aggregate more temporal dynamics to the representation making it able to capture longrange joint interactions involved in actions as well as filtering noisy motion values. Experimental results demonstrate the effectiveness of the proposed representation on 3D action recognition outperforming the state-of-the-art on NTU RGB+D 120 dataset.

show abstract

“…To prove that a good structural organization of joints is important to preserve the spatial relations of the skeleton data, we compare our approach with a baseline employing random joints order when creating the representation (i.e., the creation of the chains' order C t does not take into account any semantic meaning of adjacent joints). Moreover, we also compare with the classical skeleton image representations used by state-ofthe-art approaches [12], [13], [15], [16], [18], [19] as well as to estate-of-the-art methods on the NTU RGB+D 120 [21].…”

Section: Resultsmentioning

confidence: 99%

“…Table I presents a comparison of our approach with skeleton image representations of the literature. For the methods that have more than one "image" per representation ( [13] and [15]), we stacked them to be used as input to the network. The same was performed for our TSRJI (Stacked) approach considering the images for each reference joint (i.e., S a , ... [18] achieving 70.8% of accuracy 69.5%.…”

Section: Discussionmentioning

confidence: 99%

Skeleton Image Representation for 3D Action Recognition Based on Tree Structure and Reference Joints

Caetano

Brémond

Schwartz

2019

2019 32nd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI)

122

View full text Add to dashboard Cite

In the last years, the computer vision research community has studied on how to model temporal dynamics in videos to employ 3D human action recognition. To that end, two main baseline approaches have been researched: (i) Recurrent Neural Networks (RNNs) with Long-Short Term Memory (LSTM); and (ii) skeleton image representations used as input to a Convolutional Neural Network (CNN). Although RNN approaches present excellent results, such methods lack the ability to efficiently learn the spatial relations between the skeleton joints. On the other hand, the representations used to feed CNN approaches present the advantage of having the natural ability of learning structural information from 2D arrays (i.e., they learn spatial relations from the skeleton joints). To further improve such representations, we introduce the Tree Structure Reference Joints Image (TSRJI), a novel skeleton image representation to be used as input to CNNs. The proposed representation has the advantage of combining the use of reference joints and a tree structure skeleton. While the former incorporates different spatial relationships between the joints, the latter preserves important spatial relations by traversing a skeleton tree with a depth-first order algorithm. Experimental results demonstrate the effectiveness of the proposed representation for 3D action recognition on two datasets achieving state-of-the-art results on the recent NTU RGB+D 120 dataset.

show abstract

Action Recognition Based on Joint Trajectory Maps Using Convolutional Neural Networks

Cited by 257 publications

References 22 publications

Modified Dynamic Time Warping Based on Direction Similarity for Fast Gesture Recognition

Modified Dynamic Time Warping Based on Direction Similarity for Fast Gesture Recognition

SkeleMotion: A New Representation of Skeleton Joint Sequences based on Motion Information for 3D Action Recognition

Skeleton Image Representation for 3D Action Recognition Based on Tree Structure and Reference Joints

Contact Info

Product

Resources

About