Trajectory-aware three-stream CNN for video action recognition

Weng, Zhengkui; Guan, Yuanyuan

doi:10.1117/1.jei.28.2.021004

Cited by 5 publications

(3 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These pre-computed, feature-based classification approaches are typically constructed as multiple streams to encode spatial and temporal features [18]- [21]. The commonly used motion features that have been pre-computed include motion history/energy images [22]- [24], dynamic images [25], [26], and joint trajectory maps [27]- [29] in multistream neural networks. However, a significant challenge is the effective communication between streams to exchange information for the acquisition of multi-modal spatio-temporal representations.…”

Section: Related Workmentioning

confidence: 99%

Integration of Feature and Decision Fusion With Deep Learning Architectures for Video Classification

Kiziltepe,

Gan,

Escobar

2024

IEEE Access

View full text Add to dashboard Cite

Information fusion is frequently employed to integrate diverse inputs, including sensory data, features, or decisions, in order to leverage the advantageous relationships among various features and classifiers. This paper presents a novel approach for video classification using deep learning architectures, including ConvLSTM and vision transformer based fusion architectures, which incorporates the combination of spatial and temporal features, along with the utilisation of decision fusion at multiple levels. The proposed vision transformer based method uses a 3D CNN to extract spatio-temporal information and different attention mechanisms to pay attention to essential features for action recognition and thus learns spatiotemporal dependencies effectively. The effectiveness of the methods proposed in this paper is validated through empirical evaluations conducted on two well-known video classification datasets, namely UCF-101 and KTH. The experimental findings indicate that the utilisation of both spatial and temporal features is essential, with the superior performance gained by using temporal features as the primary source of features in conjunction with two types of distinct spatial features when compared to other configurations. The multilevel decision fusion approach proposed in this study produces results comparable to those of feature fusion methods while offering the advantage of reduced memory requirements and computational costs. The fusion of RGB, HOG, and optical flow representations has demonstrated the best performance compared to other fusion methods examined in this study. It has also been demonstrated that the vision transformer based approaches significantly outperformed the ConvLSTM based approaches. Furthermore, an ablation study was conducted to compare the performances of vision transformer based feature fusion approaches for enhancing the performance of video classification.

show abstract

Section: Related Workmentioning

confidence: 99%

Integration of Feature and Decision Fusion With Deep Learning Architectures for Video Classification

Kiziltepe,

Gan,

Escobar

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Human action recognition, 18 specifically related to violence detection in videos, is a basic task that involves extracting the human behavior features. At present, the mainstream technology in this field is based on deep learning.…”

Section: Related Workmentioning

confidence: 99%

VioNets: efficient multi-modal fusion method based on bidirectional gate recurrent unit and cross-attention graph convolutional network for video violence detection

Liang

Xiao

2023

J. Electron. Imag.

View full text Add to dashboard Cite

.Recent research on video violence detection has made great progress with the development of multi-modal fusion techniques. However, existing approaches still pose huge challenges in real-time violence detection due to fusion features being fixed and not able to be further fine tuned. We aim to address this challenge by exploring the multi-modal fusion method for analyzing multiple modal violent information, i.e., audio, optical flows, and RGB images. We propose a unified network called VioNets, which contains both a cross-attention graph convolutional network (GCN) module and a bidirectional gate recurrent unit (Bi-GRU) module for fusing different modalities of information. First, the cross-attention GCN module is utilized to extract the cross-modal spatial–temporal features. The Bi-GRU module is then applied to accurately capture both past and future context features for each time step of the single-modal features. As a result, the model retains important single-modal information in the extracted features while using the cross-modal features to improve the detection accuracy. Experiments conducted on the XD-Violence dataset show that the proposed method achieves an average precision of 80.59% and an inference time of 0.16 s with 1.82 M parameters.

show abstract

“…At present, most of the researches in the field of activity recognition focus on how to recognize human activity in a complete image sequence [1][2][3]. However, in practical applications, people is more desirable that the intelligent system can warn of the potential risks in advance so as to stop dangerous acts before they cause serious damage, rather than just recognizing the dangerous activity or detecting the damage caused by it.…”

Section: Introductionmentioning

confidence: 99%

Human activity prediction using saliency-aware motion enhancement and weighted LSTM network

Weng

Zhang

2021

J Image Video Proc.

Self Cite

View full text Add to dashboard Cite

In recent years, great progress has been made in recognizing human activities in complete image sequences. However, predicting human activity earlier in a video is still a challenging task. In this paper, a novel framework named weighted long short-term memory network (WLSTM) with saliency-aware motion enhancement (SME) is proposed for video activity prediction. First, a boundary-prior based motion segmentation method is introduced to use shortest geodesic distance in an undirected weighted graph. Next, a dynamic contrast segmentation strategy is proposed to segment the moving object in a complex environment. Then, the SME is constructed to enhance the moving object by suppressing irrelevant background in each frame. Moreover, an effective long-range attention mechanism is designed to further deal with the long-term dependency of complex non-periodic activities by automatically focusing more on the semantic critical frames instead of processing all sampled frames equally. Thus, the learned weights can highlight the discriminative frames and reduce the temporal redundancy. Finally, we evaluate our framework on the UT-Interaction and sub-JHMDB datasets. The experimental results show that WLSTM with SME statistically outperforms a number of state-of-the-art methods on both datasets.

show abstract

Trajectory-aware three-stream CNN for video action recognition

Cited by 5 publications

References 30 publications

Integration of Feature and Decision Fusion With Deep Learning Architectures for Video Classification

Integration of Feature and Decision Fusion With Deep Learning Architectures for Video Classification

VioNets: efficient multi-modal fusion method based on bidirectional gate recurrent unit and cross-attention graph convolutional network for video violence detection

Human activity prediction using saliency-aware motion enhancement and weighted LSTM network

Contact Info

Product

Resources

About