Multi‐stream 3D CNN structure for human action recognition trained by limited data

Abstract

Here, the authors proposed a solution to improve the training performance in limited training data case for human action recognition. The authors proposed three different convolutional neural network (CNN) architectures for this purpose. At first, the authors generated four different channels of information by optical flows and gradients in the horizontal and vertical directions from each frame to apply to three-dimensional (3D) CNNs. Then, the authors proposed three architectures, which are single-stream, two-stream, and four-stream 3D CNNs. In the single-stream model, the authors applied four channels of information from each frame to a single stream. In the two-stream architecture, the authors applied optical flow-x and optical flow-y into one stream and gradient-x and gradient-y to another stream. In the four-stream architecture, the authors applied each one of the information channels to four separate streams. Evaluating the architectures in an action recognition system, the system was assessed on IXMAS, a data set which has been recorded simultaneously by five cameras. The authors showed that the results of four-stream architecture were better than other architectures, achieving 87.5, 91.66, 91.11, 88.05, and 81.94% recognition rates for cameras 0-4, respectively, using four-stream structure (88.05% recognition rate in average).

[Retracted] Human Action Recognition in Smart Cultural Tourism Based on Fusion Techniques of Virtual Reality and SOM Neural Network

“…As shown in Figure 4, after inputting the action feature vector obtained from the action video frame sequence into the self-organizing mapping network (SOM), through competitive learning, the trained weights of neurons in the competitive layer can be obtained [23]. e Euclidean distance between the weights of different neurons in the competitive layer of the feature vector and SOM neural network is obtained [24][25][26][27][28][29][30][31]. According to the Euclidean distance, the feature vectors are classified into different neurons; the last neuron is traversed to find out the nearest feature vector of each non-control neuron as the key frame [32][33][34][35][36][37][38][39].…”

Feature Extraction Method of Snowboard Starting Action Using Vision Sensor Image Processing

“…is method segmented the image in advance and then extracted the action features, so as to obtain the Gaussian distribution model of the action image background and realize the extraction of human action, but the feature extraction clustering of this method is poor. Literature [9] proposed a human motion recognition method based on 3D CNN, which encodes the motion information to recognize and extract the motion. However, the stability of motion feature extraction is low.…”

Binary dense sift flow based two stream CNN for human action recognition

“…The BinaryDataCost function of (6) consists of the sum of the delta function. As shown in (7), calculating the delta function simply yields 0 and 1 outputs, and is faster than calculating L1 norm. This modified data term reduces the computational cost than conventional SIFT flow but maintains the performance of human action recognition.…”

“…However, real videos of human actions do not satisfy the above assumption, especially in the ADAS system and in autonomous vehicles, which results in the optical flow-based two-stream CNN having low performance for human action recognition. To overcome this limitation of optical flow, many researchers have modified the two-stream CNN network model [13,37] with a long short-term memory (LSTM) [12,45], a complicated pooling layer [14,42,46,47,49], or more than three stream networks [7,28,39,44]. Although the modifications improve their action recognition accuracy, the modified optical-flow two-stream CNNs have been found in recent research to have the same limitations [31].…”