Multi-scale contrast and relative motion-based key frame extraction

Abstract

The huge amount of video data available these days requires effective management techniques for storage, indexing, and retrieval. Video summarization, a method to manage video data, provides concise versions of the videos for efficient browsing and retrieval. Key frame extraction is a form of video summarization which selects only the most salient frames from a given video. Since the automatic semantic understanding of the video contents is not possible so far, most of the existing works employ low level index features for extracting key frames. However, the usage of low level features results in loss of semantic details, thus leading to a semantic gap. In this context, the saliency-based user attention modeling technique can be used to bridge this semantic gap. In this paper, a key frame extraction scheme based on a visual attention mechanism is proposed. The proposed scheme builds static visual attention method based on multi-scale contrast instead of usual color contrast. The dynamic visual attention model is developed based on novel relative motion intensity and relative motion orientation. An efficient fusion scheme for combining three visual attention values is then proposed. A flexible technique is then used for key frame extraction. The experimental results demonstrate that the proposed mechanism provides excellent results as compared to the some of the other prominent techniques in the literature.

SalSum: Saliency-based Video Summarization using Generative Adversarial Networks

“…In addition, the saliency maps are derived and the final summaries are obtained using the Earth Mover's Distance and histogram intersection similarity between them and the ground truth. In [33] the considered features included color, multi-scale contrast extracted by a Gaussian pyramid instead of usual color contrast to create static visual attention, motion obtained through optical flow to create a dynamic visual attention and relative motion orientation. Afterwards, an efficient fusion based on these three attention values takes place and keyframes are extracted by using the Euclidean similarity.…”

Keyframe Extraction Algorithm for Continuous Sign-Language Videos Using Angular Displacement and Sequence Check Metrics

“…Te various keyframe extraction paradigms utilized in video-related computer vision tasks are cluster methods [9,10], motion energy-based methods [11], sequence methods [12][13][14][15][16], and machine learning methods [17,18]. Diferent sequential approaches and machine learning methods are the most acceptable techniques used in keyframe extraction from continuous sign-language videos.…”

Various frameworks for integrating image and video streams for spatiotemporal information learning employing 2D-3D Residual networks for human action recognition

“…Depending on the clip's content complexity and keyframe extraction method, one or more keyframes can be extracted from a single clip [36]. The commonly used keyframe extraction methods include: (1) keyframe extraction method based on video lens [37] (2) clustering based keyframe extraction method [38] (3) keyframe extraction method based on motion features [39](4) keyframe extraction method based on content analysis [40,41]. The keyframe extraction approaches considerably reduce the temporal complexity of action recognition and increase the recognition model's performance.…”