Data-level information enhancement: Motion-patch-based Siamese Convolutional Neural Networks for human activity recognition in videos

Zhang, Yujia; Po, Lai-Man; Liu, Mengyang; Rehman, Yasar Abbas Ur; Ou, Weifeng; Zhao, Yuzhi

doi:10.1016/j.eswa.2020.113203

Cited by 19 publications

(10 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the Individual strategy, the features of the context stream and the saliency stream of ASNet were trained individually with the same fully connected layer (the classify layer) and the predictive scores of each stream were averaged for the final classification. Other strategies are the same as [ 54 , 72 ]. The comparison results can be seen in Table 3 , where we report the accuracy on the first split of UCF-101 and HMDB-51 with 16 f clips.…”

Section: Methodsmentioning

confidence: 99%

“…In addition, we observed that as the network capabilities increase (i.e., deeper), the performance of ASNet will be better. In this section, we compare five different fusion strategies referring to [54,72], such as Individual, Sum, Concatenation, Convolution and Multiply. The fusion layer is injected after the last convolutional layer since the features at that point are highly informative following [72].…”

Section: Asnet With Different Backbonesmentioning

confidence: 99%

“…For action recognition, Tu et al [ 53 ] proposed a human-related multi-stream CNN architecture with six CNN branches, in which the human detection algorithm is applied to salience detection of the saliency stream. Zhang et al [ 54 ] proposed a Siamese Neural Network guided by motion patches based on optical flow to enhance motion information. Jiang et al [ 4 ] proposed the use of Mask R-CNN detection to establish a saliency attention layer to eliminate CNN’s intra-frame redundancy.…”

Section: Related Workmentioning

confidence: 99%

“…The cropping process of the salient patch is formulated as a Markov decision process, which uses a weakly supervised learning method to crop the patch without extra labels. The reward of SPA is calculated based on the loss of the fully connected neural network classifier

, which uses concatenated features [ 4 , 54 ] from the context network and saliency network as shown in Figure 2 .…”

Section: Asnet Frameworkmentioning

confidence: 99%

See 3 more Smart Citations

ASNet: Auto-Augmented Siamese Neural Network for Action Recognition

Zhang

Xiong

et al. 2021

Sensors

Self Cite

View full text Add to dashboard Cite

Human action recognition methods in videos based on deep convolutional neural networks usually use random cropping or its variants for data augmentation. However, this traditional data augmentation approach may generate many non-informative samples (video patches covering only a small part of the foreground or only the background) that are not related to a specific action. These samples can be regarded as noisy samples with incorrect labels, which reduces the overall action recognition performance. In this paper, we attempt to mitigate the impact of noisy samples by proposing an Auto-augmented Siamese Neural Network (ASNet). In this framework, we propose backpropagating salient patches and randomly cropped samples in the same iteration to perform gradient compensation to alleviate the adverse gradient effects of non-informative samples. Salient patches refer to the samples containing critical information for human action recognition. The generation of salient patches is formulated as a Markov decision process, and a reinforcement learning agent called SPA (Salient Patch Agent) is introduced to extract patches in a weakly supervised manner without extra labels. Extensive experiments were conducted on two well-known datasets UCF-101 and HMDB-51 to verify the effectiveness of the proposed SPA and ASNet.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Asnet With Different Backbonesmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

, which uses concatenated features [ 4 , 54 ] from the context network and saliency network as shown in Figure 2 .…”

Section: Asnet Frameworkmentioning

confidence: 99%

See 2 more Smart Citations

ASNet: Auto-Augmented Siamese Neural Network for Action Recognition

Zhang

Xiong

et al. 2021

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…CNNs can also be used to determine the similarities between images. Siamese neural networks can be used to determine similarities between images, and they are widely used in applications, such as patch matching [10], human activity recognition [20], and tracking [18]. Siamese neural networks have a two-tower structure [15], and they use two CNN branches with similar structures and shared weight parameters to learn the image features and determine the similarities between two images.…”

Section: Siamese Networkmentioning

confidence: 99%

Object matching between visible and infrared images using a Siamese network

Chen

et al. 2021

Appl Intell

View full text Add to dashboard Cite

In this study, we propose a method for object matching between visible and infrared images. We consider object matching between visible and infrared images as a computational patch-matching problem, and the main solution to this problem involves calculating the similarities of the relationships between the objects in the target and search images. Therefore, in this study, we propose a Siamese neural network, which takes a pair of visible and infrared images as the input. Our proposed Siamese network comprises a convolutional neural network (CNN) to ensure the effective extraction of features from visible and infrared images. The CNN comprises convolutional and pooling layers without padding. By calculating the cross-correlation of the objects in the visible image and those in the entire infrared image, we regard the parts with the highest similarity as the matched targets. During the training process, we use focal loss to solve the problem of the imbalance between the positive and negative samples in the dataset, after which we use interpolation to determine the locations of the target patches in the infrared images. We then conduct experiments on different classes of targets, and the results demonstrate that our proposed approach achieves greater accuracy and precision than other methods.

show abstract