Exploiting semantics on external resources to gather visual examples for video retrieval

Abstract

With the huge and ever rising amount of video content available on the Web, there is a need to facilitate video retrieval functionalities on very large collections. Most of the current Web video retrieval systems rely on manual textual annotations to provide keyword-based search interfaces. These systems have to face the problems that users are often reticent to provide annotations, and that the quality of such annotations is questionable in many cases. An alternative commonly used approach is to ask the user for an image example, and exploit the low-level features of the image to find video content whose keyframes are similar to the image. In this case, the main limitation is the so-called semantic gap, which consists of the fact that low-level image features often do not match with the real semantics of the videos. Moreover, this approach may be a burden to the user, as it requires finding and providing the system with relevant visual examples. Aiming to address this limitation, in this paper, we present a hybrid video retrieval technique that automatically obtains visual examples by performing textual searches on external knowledge sources, such as DBpedia, Flickr and Google Images, which have different coverage and structure characteristics. Our approach exploits the semantics underlying the above knowledge sources to address the semantic gap problem. We have conducted evaluations to assess the quality of visual examples retrieved from the above external knowledge sources.

Semantic Based Video Retrieval System: Survey

“…Vallet et al [76] are designed system that is exploited external knowledge to provide visual examples related to a user search query which is employed as search inputs for low level feature retrieval models. Different external knowledge is employed in the proposed system: DBpedia (a highly structured), Flickr (a semi-structured) and Google Images (no metadata structure) which have different characteristics.…”

Combining heterogeneous sources in an interactive multimedia content retrieval model

“…Hu et al (2011) describe a music retrieval system, which searches for similar audio elements using text and then queries a content-based music retrieval system. The approach proposed in (Vallet et al, 2012) first searches using text in the external sources DBPedia, Flicker, and Google Images, and then uses these images to retrieve video by visual content. In Hauptmann et al (2002), a video retrieval system is presented that allows voice queries.…”