YouTube Scale, Large Vocabulary Video Annotation

Abstract

As video content on the web continues to expand, it is increasingly important to properly annotate videos for effective search and mining. While the idea of annotating static imagery with keywords is relatively well known, the idea of annotating videos with natural language keywords to enhance search is an important emerging problem with great potential to improve the quality of video search. However, leveraging web-scale video datasets for automated annotation also presents new challenges and requires methods specialized for scalability and efficiency. In this chapter we review specific, state of the art techniques for video analysis, feature extraction and classification suitable for extremely large scale automated video annotation. We also review key algorithms and data structures that make truly large scale video search possible. Drawing from these observations and insights, we present a complete method for automatically augmenting keyword annotations to videos using previous annotations for a large collection of videos. Our approach is designed explicitly to scale to YouTube sized datasets and we present some experiments and analysis for keyword augmentation quality using a corpus of over 1.2 million YouTube videos. We demonstrate how the automated annotation of webscale video collections is indeed feasible, and that an approach combining visual features with existing textual annotations yields better results than unimodal models.

Describing Videos by Exploiting Temporal Structure

“…every minute, 100 hours of video are uploaded to YouTube. 1 However, if a video is poorly tagged, its utility is dramatically diminished [24]. Automatic video description generation has the potential to help improve indexing and search quality for online videos.…”

Hybrid Indexes for Spatial-Visual Search

“…First, LSH is more suitable for indexing global image descriptors (e.g., CNN) while vocabulary tree is built to index local image descriptors (e.g., SIFT). Second, LSH shows better search performance because of using the hashing technique while the vocabulary tree uses the recursive clustering to partition the space resulting in worse performance and higher inaccuracy especially when the tree becomes deeper [22].…”

High-level event recognition in unconstrained videos

“…Brezeale and Cook [17] surveyed text, video, and audio features for classifying videos into a predefined set of genres, e.g., "sports" or "comedy". Morsillo et al [94] presented a brief review that focused on efficient and scalable methods for annotating Web videos at various levels including objects, scenes, actions, and high-level events. Lavee et al [67] reviewed event modeling methods, mostly in the context of simple human activity analysis.…”