Semantic description of a video using representative frames

Abstract

Analysis of a very long video and semantically describe the contents is a challenging task in computer vision. The present approaches such as video shot detection and summarization address this problem partially while maintaining the temporal coherency. To reduce the user efforts for seeing the whole video we have introduced a new technique which combines similar content irrespective of their presence at different time instants. In this approach, we automatically identify only the representative frames corresponding to similar scenes which were captured at different instants of time. We also provide the labels of the objects that are present in the representative frames along with the compact representation for the video. We achieve the task of semantic labelling of frames in a unified framework using a deep learning framework involving pre-trained features through a convolutional neural network. We show that the proposed approach is able to address the semantic labelling effectively as justified by the results obtained for videos of different scenes captured through different modalities.