Saliency detection in MPEG and HEVC video using intra-frame and inter-frame distances

Shanableh, Tamer

doi:10.1007/s11760-015-0798-9

Cited by 8 publications

(5 citation statements)

References 27 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The authors of [36] first suggested using stepwise regression in video-based intelligent systems. Since then, it has been effectively employed in many vision-based applications, as documented in several works, including [12,37,38].…”

Section: Reducing the Feature Spacementioning

confidence: 99%

“…This work aims to leverage the useful information encapsulated by HEVC coding in the video bitstream. HEVC bitstream information in the form of features was proven to be useful in several applications, such as static video summarization [8], encoding speedup and video transcoding [9], data embedding [10], the detection of double and triple compression [11], and saliency detection [12].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Static Video Summarization Using Video Coding Features with Frame-Level Temporal Subsampling and Deep Learning

Issa

Shanableh

2023

Applied Sciences

Self Cite

View full text Add to dashboard Cite

There is an abundance of digital video content due to the cloud’s phenomenal growth and security footage; it is therefore essential to summarize these videos in data centers. This paper offers innovative approaches to the problem of key frame extraction for the purpose of video summarization. Our approach includes the extraction of feature variables from the bit streams of coded videos, followed by optional stepwise regression for dimensionality reduction. Once the features are extracted and their dimensionality is reduced, we apply innovative frame-level temporal subsampling techniques, followed by training and testing using deep learning architectures. The frame-level temporal subsampling techniques are based on cosine similarity and the PCA projections of feature vectors. We create three different learning architectures by utilizing LSTM networks, 1D-CNN networks, and random forests. The four most popular video summarization datasets, namely, TVSum, SumMe, OVP, and VSUMM, are used to evaluate the accuracy of the proposed solutions. This includes the precision, recall, F-score measures, and computational time. It is shown that the proposed solutions, when trained and tested on all subjective user summaries, achieved F-scores of 0.79, 0.74, 0.88, and 0.81, respectively, for the aforementioned datasets, showing clear improvements over prior studies.

show abstract

Section: Reducing the Feature Spacementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Static Video Summarization Using Video Coding Features with Frame-Level Temporal Subsampling and Deep Learning

Issa

Shanableh

2023

Applied Sciences

Self Cite

View full text Add to dashboard Cite

show abstract

“…We modified the coder to generate low-level features, as described in the next section. HEVC features have been successfully used in many applications, including encoding speedup [35], video transcoding [35], data embedding [36], double and triple compression detection [37] and saliency detection [38]. Eventually, all the feature variables are added to the .h5 files of the OVP and VSUMM, as illustrated in Fig.…”

Section: Table II Proposed Hevc Feature Set Per Frame a Custom Hevc D...mentioning

confidence: 99%

“…The use of stepwise regression in video-based intelligent systems was first proposed by the authors in [41]. Since then, it was successfully used with video codec as reported in [38], [42] and [43] to mention a few. In this work, we propose the use of stepwise regression to reduce the dimensionality of both HEVC and CNNs features, in which we treat the feature variables as predictors and the class labels as response variables.…”

Section: B Proposed Stepwise Regression Solutionmentioning

confidence: 99%

CNN and HEVC Video Coding Features for Static Video Summarization

Issa

Shanableh

2022

IEEE Access

Self Cite

View full text Add to dashboard Cite

This study proposes a novel solution for the detection of keyframes for static video summarization. We preprocessed the well-known video datasets by coding them using the HEVC video coding standard. During coding, 64 proposed features were generated from the coder for each frame. Additionally, we converted the original YUVs of the raw videos into RGB images and fed them into pretrained CNN networks for feature extraction. These include GoogleNet, AlexNet, Inception-ResNet-v2, and VGG16. The modified datasets are made publicly available to the research community. Before detecting keyframes in a video, it is important to identify and eliminate duplicate or similar video frames. A subset of the proposed HEVC feature set was used to identify these frames and eliminate them from the video. We also propose an elimination solution based on the sum of the absolute differences between a frame and its motion-compensated predecessor. The proposed solutions are compared with existing works based on an SIFT flow algorithm that uses CNN features. Subsequently, an optional dimensionality reduction based on stepwise regression was applied to the feature vectors prior to detecting key frames. The proposed solution is compared with existing studies that use sparse autoencoders with CNN features for dimensionality reduction. The accuracy of the proposed key-frame detection system was assessed using the positive predictive values, sensitivity, and F-scores. Combining the proposed solution with Multi-CNN features and using a random forest classifier, it was shown that the proposed solution achieved an average F-score of 0.98.

show abstract

“…Chen [23] designed a visual defect detection method based on multispectral deep CNN, wherein the CNN model explores surface defect information in images of different spectral bands with enhanced recognition abilities for complex texture background and defect features. Several other efforts have also been reported in the literature, such as a deep learning-based classification pipeline structure [24] introduced to perform preprocessing (e.g., distortion correction, segmentation, and perspective correction on electro-luminescence images), a deep CNN designed for solar cell surface defect classification, and an investigation on the influence of a few oversamples and increase in data on system accuracy [25].…”

Section: Introductionmentioning

confidence: 99%

Surface Defect Detection of Solar Cells Based on Multiscale Region Proposal Fusion Network

et al. 2021

View full text Add to dashboard Cite

Manufacturing process and human operational errors may cause small-sized defects, such as cracks, over-welding, and black edges, on solar cell surfaces. These surface defects are subtle and, therefore, difficult to observe and detect. Accurate detection and replacement of defective battery modules is necessary to ensure the energy conversion efficiency of solar cells. To improve the adaptability to the scale changes of various types of surface defects of solar cells, this study proposed a multi-feature region proposal fusion network (MF-RPN) structure to detect surface defects. In such a network, region proposals are extracted from different feature layers of convolutional neural networks. Additionally, considering that multiple aspect ratios and scale settings and the use of multiple RPNs, result in an overlap of candidate regions and lead to information redundancy, we designed a multiscale region proposal selection strategy (MRPSS) to reduce the number of region proposals and improve network accuracy. Owing to the complete learning of shallow-detail texture information and deep semantic information, our multiscale RPN fusion structure can effectively improve an object's multiscale feature extraction ability for various scales and types of surface defects of solar cells. Experimental results demonstrate that our method outperforms other methods by achieving a higher detection accuracy.

show abstract

Saliency detection in MPEG and HEVC video using intra-frame and inter-frame distances

Abstract: Experimental results revealed that a Kullback-Leibler distance of 2.14 and area under the receiveroperator curve of 0.936 are achieved.

Cited by 8 publications

References 27 publications

Static Video Summarization Using Video Coding Features with Frame-Level Temporal Subsampling and Deep Learning

Static Video Summarization Using Video Coding Features with Frame-Level Temporal Subsampling and Deep Learning

CNN and HEVC Video Coding Features for Static Video Summarization

Surface Defect Detection of Solar Cells Based on Multiscale Region Proposal Fusion Network

Contact Info

Product

Resources

About