Extracting Semantics from Multimedia Content: Challenges and Solutions

Abstract

Multimedia content accounts for over 60% of traffic in the current internet [74]. With many users willing to spend their leisure time watching videos on YouTube or browsing photos through Flickr, sifting through large multimedia collections for useful information, especially those outside of the open web, is still an open problem. The lack of effective indexes to describe the content of multimedia data is a main hurdle to multimedia search, and extracting semantics from multimedia content is the bottleneck for multimedia indexing. In this chapter, we present a review on extracting semantics from a large amount of multimedia data as a statistical learning problem. Our goal is to present the current challenges and solutions from a few different perspectives and cover a sample of related work. We start with an system overview with the five major components that extracts and uses semantic metadata: data annotation, multimedia ontology, feature representation, model learning and retrieval systems. We then present challenges for each of the five components along with their existing solutions: designing multimedia lexicons and using them for concept detection, handling multiple media sources and resolving correspondence across modalities, learning structured (generative) models to account for natural data dependency or model hidden topics, handling rare classes, leveraging unlabeled data, scaling to large amounts of training data, and finally leveraging media semantics in retrieval systems.

Managing Event Information: Modeling, Retrieval, and Applications

“…In these domains, some researchers have tried to use supervised machine learning technique [Xie and Yan, 2008] for the detection of semantic objects, themes and events. In these domains, some researchers have tried to use supervised machine learning technique [Xie and Yan, 2008] for the detection of semantic objects, themes and events.…”

Event Mining in Multimedia Streams

“…In terms of coverage, there have been a few surveys in multimedia analysis and indexing but none have discussed events in depth. There are surveys on extracting generic static semantics [151], in which event is an important but distinct subset mentioned in the passing, or on image/video retrieval [20], [118], [129], [131], [155], which can be an application for event analysis. In terms of organization, Section II provides a component overview similar to a few image/video retrieval surveys [78], [129], [131], [155]; Section IV surveys existing event detection systems grouped by common problem scope and solution components, similar to earlier system surveys [20], [118].…”