Best Frame Selection to Enhance Training Step Efficiency in Video-Based Human Action Recognition

Gharahbagh, Abdorreza Alavi; Hajihashemi, Vahid; Ferreira, Marta Campos; Machado, J.J.M.; Tavares, João Manuel R. S.

doi:10.3390/app12041830

Cited by 10 publications

(4 citation statements)

References 36 publications

(46 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In their study, three DNNs were trained using motion and spatial features, and the results were combined in a fusion step. Gharahbagh et al [32] used a combination of temporal and spatial features, such as frame differences and gradient, for the best frame selection in a HAR system to improve the training efficiency. Hajihashemi and Pakizeh [33] proposed a HAR system based on the gradient in both temporal and spatial domains.…”

Section: State Of the Artmentioning

confidence: 99%

Deep Learning Approach for Human Action Recognition Using a Time Saliency Map Based on Motion Features Considering Camera Movement and Shot in Video Image Sequences

Alavigharahbagh,

Hajihashemi,

Machado

et al. 2023

Information

Self Cite

View full text Add to dashboard Cite

In this article, a hierarchical method for action recognition based on temporal and spatial features is proposed. In current HAR methods, camera movement, sensor movement, sudden scene changes, and scene movement can increase motion feature errors and decrease accuracy. Another important aspect to take into account in a HAR method is the required computational cost. The proposed method provides a preprocessing step to address these challenges. As a preprocessing step, the method uses optical flow to detect camera movements and shots in input video image sequences. In the temporal processing block, the optical flow technique is combined with the absolute value of frame differences to obtain a time saliency map. The detection of shots, cancellation of camera movement, and the building of a time saliency map minimise movement detection errors. The time saliency map is then passed to the spatial processing block to segment the moving persons and/or objects in the scene. Because the search region for spatial processing is limited based on the temporal processing results, the computations in the spatial domain are drastically reduced. In the spatial processing block, the scene foreground is extracted in three steps: silhouette extraction, active contour segmentation, and colour segmentation. Key points are selected at the borders of the segmented foreground. The last used features are the intensity and angle of the optical flow of detected key points. Using key point features for action detection reduces the computational cost of the classification step and the required training time. Finally, the features are submitted to a Recurrent Neural Network (RNN) to recognise the involved action. The proposed method was tested using four well-known action datasets: KTH, Weizmann, HMDB51, and UCF101 datasets and its efficiency was evaluated. Since the proposed approach segments salient objects based on motion, edges, and colour features, it can be added as a preprocessing step to most current HAR systems to improve performance.

show abstract

Section: State Of the Artmentioning

confidence: 99%

Deep Learning Approach for Human Action Recognition Using a Time Saliency Map Based on Motion Features Considering Camera Movement and Shot in Video Image Sequences

Alavigharahbagh,

Hajihashemi,

Machado

et al. 2023

Information

Self Cite

View full text Add to dashboard Cite

show abstract

“…Finally, discrete Hidden Markov Models (HMMs) are applied to classify and recognise action sequences. Meanwhile, in order to effectively improve the recognition performance of the system, some researchers have adopted a keyframe-based approach to reduce the processing time of the system [19,20]. A recognition system for human action sequences was developed using traditional machine learning algorithms combined with key-frame selection.…”

Section: Traditional Machine Learning and Hand-crafted Feature-based ...mentioning

confidence: 99%

Human Action Recognition Using Key-Frame Attention-Based LSTM Networks

et al. 2023

View full text Add to dashboard Cite

Human action recognition is a classical problem in computer vision and machine learning, and the task of effectively and efficiently recognising human actions is a concern for researchers. In this paper, we propose a key-frame-based approach to human action recognition. First, we designed a key-frame attention-based LSTM network (KF-LSTM) using the attention mechanism, which can be combined with LSTM to effectively recognise human action sequences by assigning different weight scale values to give more attention to key frames. In addition, we designed a new key-frame extraction method by combining an automatic segmentation model based on the autoregressive moving average (ARMA) algorithm and the K-means clustering algorithm. This method effectively avoids the possibility of inter-frame confusion in the temporal sequence of key frames of different actions and ensures that the subsequent human action recognition task proceeds smoothly. The dataset used in the experiments was acquired with an IMU sensor-based motion capture device, and we separately extracted the motion features of each joint using a manual method and then performed collective inference.

show abstract

“…The authors verified that their method was effective at detecting key frames for action recognition. In addition, numerous deep-learning-based approaches for key-frame detection ( [13][14][15][16][17][18][19][20]) were proposed. They commonly considered the knowledge between consecutive frames to capture the correlation in a video, which is contrary to our approach.…”

Section: Key Frame Detectionmentioning

confidence: 99%

Extending Contrastive Learning to Unsupervised Redundancy Identification

Jung

Kim

2022

Applied Sciences

View full text Add to dashboard Cite

Modern deep neural network (DNN)-based approaches have delivered great performance for computer vision tasks; however, they require a massive annotation cost due to their data-hungry nature. Hence, given a fixed budget and unlabeled examples, improving the quality of examples to be annotated is a clever step to obtain good generalization of DNN. One of key issues that could hurt the quality of examples is the presence of redundancy, in which the most examples exhibit similar visual context (e.g., same background). Redundant examples barely contribute to the performance but rather require additional annotation cost. Hence, prior to the annotation process, identifying redundancy is a key step to avoid unnecessary cost. In this work, we proved that the coreset score based on cosine similarity (cossim) is effective for identifying redundant examples. This is because the collective magnitude of the gradient over redundant examples exhibits a large value compared to the others. As a result, contrastive learning first attempts to reduce the loss of redundancy. Consequently, cossim for the redundancy set exhibited a high value (low coreset score). We first viewed the redundancy identification as the gradient magnitude. In this way, we effectively removed redundant examples from two datasets (KITTI, BDD10K), resulting in a better performance in terms of detection and semantic segmentation.

show abstract

Best Frame Selection to Enhance Training Step Efficiency in Video-Based Human Action Recognition

Cited by 10 publications

References 36 publications

Deep Learning Approach for Human Action Recognition Using a Time Saliency Map Based on Motion Features Considering Camera Movement and Shot in Video Image Sequences

Deep Learning Approach for Human Action Recognition Using a Time Saliency Map Based on Motion Features Considering Camera Movement and Shot in Video Image Sequences

Human Action Recognition Using Key-Frame Attention-Based LSTM Networks

Extending Contrastive Learning to Unsupervised Redundancy Identification

Contact Info

Product

Resources

About