3D video shot boundary detection based on clustering of depth-temporal features

Abstract

This paper proposes an algorithm for automatic detection of 3D video shots with different perceptual features. The proposed algorithm is able to identify distinct three-dimensional visual scenes by detecting 3D video shot boundaries based on clustering of depth-temporal features. A combination of texture variation along the temporal dimension and depth variance is used by K-means clustering to find the stereo frames which comprised the 3D scene boundaries. An important characteristic of the proposed algorithm in comparison with others published in the literature for temporal segmentation of classic 2D video is that no thresholds are used in the decision processes neither training data sets. The experimental results show that the proposed method is capable of achieving high recall (e.g., 0.95) and precision rate (e.g., 1.0) in video sequences with both sharp and smooth 3D scene transitions.Index Terms-3D video segmentation, shot boundary detection, recall and precision rate.

Towards key-frame extraction methods for 3D video: a review

“…However, this method was applied only to the colour channels of the videos to be summarised. Ferreira et al [11] proposed an algorithm to detect 3D shot boundaries (3DSB) based on a joint depth-temporal criterion. The absolute frame difference and sum of absolute luminance histogram difference are used as the relevant measures in the temporal dimension, while in the depth dimension, the variance of depth in each frame is used.…”

“…The decision methods used to find shot boundaries can be based on static thresholds (as in Fig. 2), adaptive thresholds (thresholds depend on the statistics of the visual features used), B-splines fittings [30], support vector machines (SVM) [31] and K-means clustering [11]. The detection accuracy of SBD methods is improved by combining several visual features [32].…”

“…The feature vector including segment size, location, colour and depth. Another important feature is the depth variance associated with the temporal activity, which was used by Ferreira et al in [11] for temporal segmentation of 3D video. The average stereo disparity per frame and temporal features such as image difference and histogram difference are computed and combined in feature vectors which in turn are used by a clustering algorithm to partitioning 3D video in temporal segments.…”

“…1, the input video is segmented into video shots, mostly based on spatio-temporal criteria, but other criteria can be used such as based on motion [20,25] or the combination of the temporal and depth features [11]. More details can be found in Section 1.4.…”

“…The best performance is reached when F1 is equal to 1, while the worst occurs at 0. The recall rate, precision rate and measure F1 were used to evaluate the performance of temporal segmentation methods for 3D video in [11,23,38], while Yamasaki et al [20] only used recall and precision rates in the evaluation process. Although, these 3D SBD methods used the same evaluation metrics, the comparison of the results and performance obtained from such SBD methods is not possible because different datasets were used.…”