Show and Recall: Learning What Makes Videos Memorable

Abstract

With the explosion of video content on the Internet, there is a need for research on methods for video analysis which take human cognition into account. One such cognitive measure is memorability, or the ability to recall visual content after watching it. Prior research has looked into image memorability and shown that it is intrinsic to visual content, but the problem of modeling video memorability has not been addressed sufficiently. In this work, we develop a prediction model for video memorability, including complexities of video content in it. Detailed feature analysis reveals that the proposed method correlates well with existing findings on memorability. We also describe a novel experiment of predicting video sub-shot memorability and show that our approach improves over current memorability methods in this task. Experiments on standard datasets demonstrate that the proposed metric can achieve results on par or better than the state-of-the art methods for video summarization.

Annotating, Understanding, and Predicting Long-term Video Memorability

“…This work provided researchers, interested in computational understanding of IM and VM, with several memory tests (see [34] for an extensive overview), such as the recognition test [17,21,24] or the textual questions-based recall survey [36]. It also demonstrated that humans have an extensive long-term visual memory, which enables them to recall a great amount of images [38] and image details [6], as well as of videos [15].…”

“…", "Did you see a car on road?") [36]. The major drawback of the study comes from the use of questions instead of a classic visual recognition task.…”

“…As a result of this success, researchers have recently extended this challenge to videos. However, to the best of our knowledge, only two available studies focus on VM prediction [17,36].…”

“…The second point, closely related to the previous one, is the lack of a common definition for VM. The previous attempts to predict VM [17,36] were based on different measures of memorability. Furthermore, in comparison to images, videos have supplementary dimensions -sound and visual movement -that critically contribute to the semantic and emotional information conveyed and this makes it difficult to come up with a common definition of VM.…”

“…Regarding modelling, the previous attempts at predicting VM [17,36] highlighted several features which contribute to the prediction of VM, such as semantic, saliency and color features. But the work is far from complete and our capacity to propose efficient computational models will participate to answer the challenge of VM prediction.…”