High-Level Feature Detection from Video in TRECVid: A 5-Year Retrospective of Achievements

Abstract

Summary. *Successful and effective content-based access to digital video requires fast, accurate and scalable methods to determine the video content automatically. A variety of contemporary approaches to this rely on text taken from speech within the video, or on matching one video frame against others using low-level characteristics like colour, texture, or shapes, or on determining and matching objects appearing within the video. Possibly the most important technique, however, is one which determines the presence or absence of a high-level or semantic feature, within a video clip or shot. By utilizing dozens, hundreds or even thousands of such semantic features we can support many kinds of content-based video navigation. Critically however, this depends on being able to determine whether each feature is or is not present in a video clip. The last 5 years have seen much progress in the development of techniques to determine the presence of semantic features within video. This progress can be tracked in the annual TRECVid benchmarking activity where dozens of research groups measure the effectiveness of their techniques on common data and using an open, metrics-based approach. In this chapter we summarise the work done on the TRECVid high-level feature task, showing the progress made year-on-year. This provides a fairly comprehensive statement on where the state-of-the-art is regarding this important task, not just for one research group or for one approach, but across the spectrum. We then use this past and on-going work as a basis for highlighting the trends that are emerging in this area, and the questions which remain to be addressed before we can achieve large-scale, fast and reliable high-level feature detection on video. 4 Published in A. Divakaran (ed.), Multimedia Content Analysis, Signals and Communication Technology,

LifeLogging: Personal Big Data

“…This modern methodology facilitates an understanding of topic queries and low-level features by analysing the mapping in a semantic way. To build a largescale ontology and lexicon for semantic gap filling, large efforts have been made in activities like LSCOM (Large-Scale Concept Ontology for Multimedia), Naphade et al (2006); Kennedy and Hauptmann (2006), TRECVid, Smeaton et al (2009) and MediaMill's 101 concepts, Snoek et al (2006). Smeaton et al (2009) state that acceptable results have been achieved already within the TRECVid video retrieval evaluation framework for many cases particularly for concepts where there exists enough annotated training data.…”

“…To build a largescale ontology and lexicon for semantic gap filling, large efforts have been made in activities like LSCOM (Large-Scale Concept Ontology for Multimedia), Naphade et al (2006); Kennedy and Hauptmann (2006), TRECVid, Smeaton et al (2009) and MediaMill's 101 concepts, Snoek et al (2006). Smeaton et al (2009) state that acceptable results have been achieved already within the TRECVid video retrieval evaluation framework for many cases particularly for concepts where there exists enough annotated training data. Based on concept detection, encouraging improvement has been reported showing the efficiency and the effectiveness of concepts for higher level retrieval, Snoek et al (2006); Neo et al (2006).…”

Local Invariant Feature Tracks for High-Level Video Feature Extraction

“…High-level feature extraction results (average precision@2000 [6]) for both SIFT-based BoW features and features resulting from the concatenation of SIFT-and LIFT-based BoW are shown in Fig. 2.…”

“…Large scale video analysis for the purpose of high-level feature extraction, using local invariant features, is in most cases performed at the key-frame level [6]. Thus, the video analysis task reduces to still image analysis.…”

Semantically Smoothed Refinement for Everyday Concept Indexing

“…This requires searching through lifelogs based on content, and for this the automatic detection of semantic concepts is needed. The conventional approach to content-based indexing, as taken in the annual TRECVid benchmarking [11,12], is to annotate a collection covering both positive and negative examples of the presence of each concept and then to train a machine learning classifier to recognize the presence of the concept. This typically requires a classifier for each concept without considering inter-concept relationships or dependencies yet in reality, many concept pairs will co-occur rather than occur independently.…”