Rethinking the Evaluation of Video Summaries

Abstract

Video summarization is a technique to create a short skim of the original video while preserving the main stories/content. There exists a substantial interest in automatizing this process due to the rapid growth of the available material. The recent progress has been facilitated by public benchmark datasets, which enable easy and fair comparison of methods. Currently the established evaluation protocol is to compare the generated summary with respect to a set of reference summaries provided by the dataset. In this paper, we will provide in-depth assessment of this pipeline using two popular benchmark datasets. Surprisingly, we observe that randomly generated summaries achieve comparable or better performance to the state-of-the-art. In some cases, the random summaries outperform even the human generated summaries in leave-one-out experiments. Moreover, it turns out that the video segmentation, which is often considered as a fixed pre-processing method, has the most significant impact on the performance measure. Based on our observations, we propose alternative approaches for assessing the importance scores as well as an intuitive visualization of correlation between the estimated scoring and human annotations.

Weakly Supervised Video Summarization by Hierarchical Reinforcement Learning

“…Our experiments on two benchmark datasets showed that our hierarchical structure achieved the best performance out of all other methods known to us. In particular, evaluation using rank order statistics that was recently proposed in [13] clearly showed the superiority of our proposed method. Also, our proposal only requires a smaller number of task-level annotations to train the Manager.…”

“…However, our proposal was not capable of the transfer task because subgoals between different domains may vary a lot. Then, we evaluate the performance using the rank order statistics proposed in [13] introduced in Sec. 4.2, which is claimed to be a better evaluation metric without the effect of post-processing.…”

Realistic Video Summarization through VISIOCITY

“…A related problem is the fact that current supervised techniques are trained using a 'combined' ground truth summary, either in form of combined scores from multiple ground truth summaries or scores [4,12,37] or in form a set of ground truth selections, as in dP-PLSTM [37]. However, since there can be multiple correct answers, a reason for low consistency between user summaries [13,23] combining them into one misses out on the separate flavors captured by each of them. Combining many into one set of scores also runs the risk of giving more emphasis to 'importance' over and above other desirable characteristics of a summary like continuity, diversity etc.…”

“…Evaluation: With a desire to be comparable across techniques, almost all recent work evaluates their results using F1 score [4,12,39]. This approach of assessing a candidate summary vis-avis a ground truth summary sounds good, but it has following limitations: 1) The user summaries are themselves inconsistent with each other, as already noted above [13,23]. As a workaround, the assessment is done with respect to the nearest neighbor [10,30].…”

“…The number of right answers can be large, especially in case of long videos and a good candidate may get a low score just because it was not fortunate to have a matching user summary. Furthermore, F1 seems to be good to measure the 'closeness' with a user summary, but the numbers can be deceptive as it is affected by the segmentation used as a post processing step in typical video summarization pipeline [23]. F1 is also not well suited to measure other characteristics of a summary like continuity or diversity.…”