Towards interactive, intelligent, and integrated multimedia analytics

Abstract

Abstract-The size and importance of visual multimedia collections grew rapidly over the last years, creating a need for sophisticated multimedia analytics systems enabling large-scale, interactive, and insightful analysis. These systems need to integrate the human's natural expertise in analyzing multimedia with the machine's ability to process large-scale data. The paper starts off with a comprehensive overview of representation, learning, and interaction techniques from both the human's and the machine's point of view. To this end, hundreds of references from the related disciplines (visual analytics, information visualization, computer vision, multimedia information retrieval) have been surveyed. Based on the survey, a novel general multimedia analytics model is synthesized. In the model, the need for semantic navigation of the collection is emphasized and multimedia analytics tasks are placed on the explorationsearch axis. The axis is composed of both exploration and search in a certain proportion which changes as the analyst progresses towards insight. Categorization is proposed as a suitable umbrella task realizing the exploration-search axis in the model. Finally, the pragmatic gap, defined as the difference between the tight machine categorization model and the flexible human categorization model is identified as a crucial multimedia analytics topic.Index Terms-Multimedia (image/video/music) visualization, machine learning.

Ten Research Questions for Scalable Multimedia Analytics

“…3. The exploration-search axis with example multimedia analytics tasks [13] feature representation satisfying relevance criteria in a wider range of analytics tasks. For example, a particular video search query may require a simple queryby-example matching based on low-level visual features, while utilizing concept detectors or automatic speech recognition may yield better results in other cases [10].…”

“…A large body of work related to the constituent parts of multimedia analytics has been surveyed by Zahálka and Worring [13]. The objective of multimedia analytics is user insight, a complex understanding of the analyzed data accumulated over time using all or most of the relevant data at hand [8].…”

“…Nevertheless, all multimedia analytics tasks have a key common characteristic: they consist of a certain proportion of exploration and search. Hence, an exploration-search axis, shown in Figure 3, has been proposed as the task model for multimedia analytics [13]. The analysts tilt back and forth on this axis during their quest for insight, and hence a multimedia analytics system should support the entire axis.…”

“…The analysts tilt back and forth on this axis during their quest for insight, and hence a multimedia analytics system should support the entire axis. Analytic categorization, i.e., maintaining a set of analyst-defined categories based on the semantic and metadata content of the multimedia collection and updating it as the analyst progresses towards insight, has been proposed as an umbrella task for the exploration-search axis [13].…”

“…Yet the semantic gap is as prevalent at the (sub-) collection level at which it has hardly been addressed. The pragmatic gap, defined as the gap between the highly adaptable mental categorical model of the user on the one side and the strict, bounded definition of categorization in the machine world on the other, comes into play when exploring a multimedia collection in its context [13]. In order to close the pragmatic gap, the data model of the analytics system must fully adapt to user intent and understanding, as it varies over the duration of the analytics process, so that it accurately represents the view of the user at each time.…”