Multimedia Pivot Tables for Multimedia Analytics on Image Collections

Abstract

Abstract-We propose a multimedia analytics solution for getting insight in image collections by extending the powerful analytic capabilities of pivot tables, found in the ubiquitous spreadsheets, to multimedia. We formalize the concept of multimedia pivot tables and give design rules and methods for the multimodal summarization, structuring, and browsing of the collection based on these tables, all optimized to support an analyst in getting structural and conclusive insights. Our proposed solution provides truly interactive analytics on the visual content of image collections through concept detection results, as well as tags, geolocation, time and other metadata. We have performed user experiments with novice users on a dataset from Flickr to improve the initial design and with expert users in marketing and multimedia analysis on two domain specific datasets collected from Instagram. The results show that analysts are indeed capable of deriving structural and conclusive insights using the proposed multimedia analytics solution. On our website videos of the system in action are available.

II-20: Intelligent and pragmatic analytic categorization of image collections

“…They can be enhanced so as to convey collection structure [3,30,45]. Beyond grids, there are many other visual metaphors for image collections, such as spreadsheet-based that integrate the image content tightly with metadata [7,19,39], semantic-navigation-based that allow the user to pursue threads of interest, often semantic [4,8], or even metadata-driven [37,40]. The rapid serial visual presentation (RSVP) approaches present images dynamically, flashing images or video clips in a fast-paced manner, with the user providing a simple, rapid response [14,35,36].…”

“…There are approaches that incorporate interactive model building to cover a wider range of the exploration-search axis. To advance the analytic session, they usually make use of a rich set of filters on the data (e.g., [3,19,20,38,39]), an interactive (multimodal) learning model (e.g., [15,43]), or a combination of both. Whilst these techniques go beyond mere search, on the exploration-search axis, they tend to lean towards search anyway: they simply fetch what the users are looking for or what they found relevant previously.…”

Interactive Search and Exploration in Discussion Forums Using Multimodal Embeddings

“…Multimedia analytics systems, such as Multimedia Pivot Tables [31], ICLIC [28], and Blackthorn [38] facilitate search and exploration in large collections of multimedia data as well as interactive multimodal learning. Blackthorn, for example, compresses semantic information from the visual and text domain and learns user preferences on the fly from the interactions with the system in a relevance feedback framework.…”

Comparing Personal Image Collections with PICTuReVis

“…MediaTable [dRWvW10] visualizes all image properties in one big table. Multimedia Pivot Tables [WKZ16] split the collection into rows and use columns to show the distributions of the subsets over the other dimensions, such as textual description, location, time, and concept. Microsoft PivotViewer [Mic] organizes images in a zoomable histogram.…”