The Semantic Pathfinder: Using an Authoring Metaphor for Generic Multimedia Indexing

Abstract

This paper presents the semantic pathfinder architecture for generic indexing of multimedia archives. The semantic pathfinder extracts semantic concepts from video by exploring different paths through three consecutive analysis steps, which we derive from the observation that produced video is the result of an authoring-driven process. We exploit this authoring metaphor for machine-driven understanding. The pathfinder starts with the content analysis step. In this analysis step, we follow a data-driven approach of indexing semantics. The style analysis step is the second analysis step. Here, we tackle the indexing problem by viewing a video from the perspective of production. Finally, in the context analysis step, we view semantics in context. The virtue of the semantic pathfinder is its ability to learn the best path of analysis steps on a per-concept basis. To show the generality of this novel indexing approach, we develop detectors for a lexicon of 32 concepts and we evaluate the semantic pathfinder against the 2004 NIST TRECVID video retrieval benchmark, using a news archive of 64 hours. Top ranking performance in the semantic concept detection task indicates the merit of the semantic pathfinder for generic indexing of multimedia archives.

A generic framework for semantic video indexing based on visual concepts/contexts detection

“…Many techniques have been developed and several image/video indexing systems have been built [1,13,38,46]. Early systems, such as Informedia [13,14] and MARVEL multimedia analysis engine [37], proposed to combine the video modalities in order to enhance the video interpretation.…”

“…However, the majority of these systems have mainly focused on visual content processing. The semantic pathfinder [46] explored different paths through three consecutive analysis steps: content analysis, style analysis, and context analysis. In addition, other works have focused on including spatio-temporal information into visual content processing in order to detect moving objects and events [23,53,56].…”

“…The average execution times per key-frames for this process within TRECVID 2010 benchmark (146788 key-frame) took about 22.235 s on using a WINDOWS PC with 980 MHZ INTEL(R) CORE(TM) 2 CPU and 2 GB RAM. This large computing time versus the semantic Pathfinder [46] (about 1 s), invites us to parallelize the concept detection process or to use a GPU computing. However, within the refinement process, the fuzzy role are inferred simultaneously.…”

Towards User Transparent Parallel Multimedia Computing on GPU-Clusters

“…Notably, Parallel-Horus was applied in earlier NIST TRECVID benchmark evaluations for content-based video retrieval, and played a crucial role in achieving top-ranking results in a field of strong international competitors [1,7]. Moreover, recent extensions to ParallelHorus, that allow for services-based distributed multimedia computing, have been applied successfully in large-scale distributed systems, involving hundreds of massively communicating compute resources covering the entire globe [7].…”

“…The application and research domain of Multimedia Content Analysis (MMCA) investigates new methods of image and video processing, to arrive at automated techniques of extracting knowledge from multimedia data. In part, research in the MMCA domain is driven by requirements of emerging applications, ranging from the automatic comparison of forensic video evidence, to searching publicly available digital television archives, and real-time analysis of video data obtained from surveillance cameras in public locations [1].…”