Integrating multi-modal content analysis and hyperbolic visualization for large-scale news video retrieval and exploration

Abstract

In this paper, we have developed a novel scheme to achieve more effective analysis, retrieval and exploration of large-scale news video collections by performing multi-modal video content analysis and synchronization. First, automatic keyword extraction is performed on news closed captions and audio channels to detect the most interesting news topics (i.e., keywords for news topic interpretation), and the associations among these news topics (i.e., contextual relationships among the news topics) are further determined according to their co-occurrence probabilities. Second, visual semantic items, such as human faces, text captions, video concepts, are extracted automatically by using our semantic video analysis techniques. The news topics are automatically synchronized with the most relevant visual semantic items. In addition, an interestingness weight is assigned for each news topic to characterize its importance. Finally, a novel hyperbolic visualization scheme is incorporated to visualize large-scale news topics according to their associations and interestingness. With a better global overview of large-scale news video collections, users can specify their queries more precisely and explore large-scale news video collections interactively. Our experiments on large-scale news video collections have provided very positive results.

Multimedia Pivot Tables for Multimedia Analytics on Image Collections

“…Canopy [3] is an advanced system combining text analysis, various visualizations, and visual similarity based matching to explore visual collections. Hierarchical visualizations for understanding news data, based on analysis of closed-captions, are presented in [27] using hyperbolic trees as visualization to explore the hierarchical structure of the data. Concept detectors are used in both [46] and [9].…”

Insight in Image Collections by Multimedia Pivot Tables

“…Canopy [1] is an advanced system combining text analysis, various visualizations and visual similarity based matching to explore visual collections. Hierarchical visualizations for understanding news data, based on analysis of closed-captions, are presented in [17] using hyperbolic trees as visualization to explore the hierarchical structure of the data. Concept detectors are used in both [30] and [5].…”

Scalable video visual analytics

“…Due to their integration of human and machine analysis, some of these approaches can be considered as video visual analytics. Among them are MediaMill, 31,32 the Informedia project, 33,34 and the works of Luo et al 35,36 In contrast to these approaches, we put more emphasis on the analytical discourse than on the pure retrieval of particular parts of the video stream.…”