SkeletonNet: Mining Deep Part Features for 3-D Action Recognition

Ke, Qiuhong; An, Senjian; Bennamoun, Mohammed; Sohel, Ferdous; Boussaid, Farid

doi:10.1109/lsp.2017.2690339

Cited by 149 publications

(75 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…CS CV SkeletonNet(CNN) [63] 75.94% 81.16% JDM+CNN [64] 76.20% 82.30% Clips+CNN+MTLN [65] 79.57% 84.83% Enhanced Visualization+CNN [66] 80.03% 87.21% HCN [67] 86.5% 91.1% TCN + TTN [68] 77.55% 84.25% STGCN [69] 81.5% 88.3% PB-GCN [70] 87.5% 93.2% 1 Layer PLSTM [29] 62.05% 69.40% 2 Layer PLSTM [29] 62.93% 70.27% JL d+RNN [71] 70.26% 82.39% STA-LSTM [72] 73.40% 81.20% Pose conditioned STA-LSTM [73] 77.10% 84.50% 1 Layer RNN (reported in [29] are present in the scene, the skeleton identity captured by the Kinect sensor may be changed over time. Therefore, an alignment process was first applied to keep the same skeleton saved in the same data array over time.…”

Section: Methodsmentioning

confidence: 99%

Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN

Li¹,

Li²,

Cook³

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

704

421

View full text Add to dashboard Cite

Recurrent neural networks (RNNs) are known to be difficult to train due to the gradient vanishing and exploding problems and thus difficult to learn long-term patterns and construct deep networks. To address these problems, this paper proposes a new type of RNNs with the recurrent connection formulated as Hadamard product, referred to as independently recurrent neural network (IndRNN), where neurons in the same layer are independent of each other and connected across layers. The gradient vanishing and exploding problems are solved in IndRNN, and thus long-term dependencies can be learned. Moreover, an IndRNN can work with non-saturated activation functions such as ReLU (rectified linear unit) and be still trained robustly. Different deeper IndRNN architectures, including the basic stacked IndRNN, residual IndRNN and densely connected IndRNN, have been investigated, all of which can be much deeper than the existing RNNs. Furthermore, IndRNN reduces the computation at each time step and can be over 10 times faster than the commonly used Long short-term memory (LSTM). Experimental results have shown that the proposed IndRNN is able to process very long sequences and construct very deep networks. Better performances have been achieved on various tasks with IndRNNs compared with the traditional RNN and LSTM.unit (GRU) [15] have been proposed to address the gradient problems. However, the use of the hyperbolic tangent and the sigmoid functions as the activation function in these variants results in gradient decay over layers. Consequently, construction and training of a deep LSTM or GRU based RNN network is practically difficult.On the other hand, the existing RNN models share the same component σ(Wx t + Uh t−1 + b) in (1), where the recurrent connection connects all the neurons through time. This makes it hard to interpret and understand the roles of each individual neuron (e.g., what patterns each neuron responds to) without considering the others. Moreover, with the recurrent connections, matrix product is performed at each time step and the computation cannot be easily paralleled, leading to a very time-consuming process when dealing with long sequences.In this paper, we propose a new type of RNN, referred to as independently recurrent neural network (IndRNN). In the proposed IndRNN, the recurrent inputs are processed with the Hadamard product as h t = σ(Wx t +u h t−1 +b). This provides a number of advantages over the traditional RNNs including:• Able to process longer sequences: the gradient vanishing and exploding problem is solved by regulating the recurrent weights, and long-term memory can be kept in order to process long sequences. Experiments have demonstrated that an IndRNN can well process sequences over 5000 steps. • Able to construct deeper networks: multiple layers of IndRNNs can be efficiently stacked, especially with skip-connection and dense connection, to increase the depth of the network. An example of 21-layer residual IndRNN and deep densely connected In-dRNN are demonstrated in the experiments. ...

show abstract

Section: Methodsmentioning

confidence: 99%

Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN

Li¹,

Li²,

Cook³

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

704

421

View full text Add to dashboard Cite

show abstract

“…Joint-based methods model the positions and motion of the joints (either individual or a combination) using the coordinates of the joints extracted by the OpenNI tracking framework. For instance, a reference joint may be used and the coordinates of other joints are defined relative to the reference joint [10,17,20,21], or the joint orientations may be computed relative to a fixed coordinate system and used to represent the human pose [66], etc. For the body part based methods, the human body parts are used to model the human's articulated system.…”

Section: Related Workmentioning

confidence: 99%

“…Thus, a human skeleton is a point of the Lie group SE(3) × · · · × SE(3) where each action corresponds to a unique evolution of such a point in time. The approach of Ke et al [20] relies on both body parts and body joints. The human skeleton model was divided into 5 body parts.…”

Section: Related Workmentioning

confidence: 99%

“…Recent human action recognition papers favour deep learning techniques to perform human action recognition. Apart from the CNN-based approaches [20,77], Recurrent Neural Networks (RNNs) have also been popular [18,27,35,38,65,[67][68][69][70]. Since Long Short-term Memory (LSTM) [78] can model temporal dependencies as RNNs and even capture the co-occurrences of human joints, LSTM networks have also been a popular choice in human action recognition [31,33,36,37,73].…”

Section: Related Workmentioning

confidence: 99%

“…The algorithm used for computing the 3D joint positions of the human skeletal model by the Kinect toolkit is based on the human skeleton tracking framework (OpenNI) of Shotton et al [16]. In addition to the availability of the real-time depth video stream, this tracking framework also opens up the research area of skeleton-based human action recognition [17][18][19][20][21][22].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Comparative Review of Recent Kinect-Based Action Recognition Algorithms

Wang

Huynh

Koniusz

2020

IEEE Trans. on Image Process.

210

View full text Add to dashboard Cite

Video-based human action recognition is currently one of the most active research areas in computer vision. Various research studies indicate that the performance of action recognition is highly dependent on the type of features being extracted and how the actions are represented. Since the release of the Kinect camera, a large number of Kinect-based human action recognition techniques have been proposed in the literature. However, there still does not exist a thorough comparison of these Kinect-based techniques under the grouping of feature types, such as handcrafted versus deep learning features and depthbased versus skeleton-based features. In this paper, we analyze and compare ten recent Kinect-based algorithms for both crosssubject action recognition and cross-view action recognition using six benchmark datasets. In addition, we have implemented and improved some of these techniques and included their variants in the comparison. Our experiments show that the majority of methods perform better on cross-subject action recognition than cross-view action recognition, that skeleton-based features are more robust for cross-view recognition than depth-based features, and that deep learning features are suitable for large datasets.

show abstract