I-SEARCH: A Unified Framework for Multimodal Search and Retrieval

Abstract

Abstract. In this article, a unified framework for multimodal search and retrieval is introduced. The framework is an outcome of the research that took place within the I-SEARCH European Project. The proposed system covers all aspects of a search and retrieval process, namely low-level descriptor extraction, indexing, query formulation, retrieval and visualisation of the search results. All I-SEARCH components advance the state of the art in the corresponding scientific fields. The I-SEARCH multimodal search engine is dynamically adapted to end-user's devices, which can vary from a simple mobile phone to a high-performance PC.

Multimodal human-machine interface devices in the cloud

“…Of course, a lot of effort is being made to integrate various elements of interaction and develop new protocols and architectures that allow their use [2,5,8]. In fact, the W3C has a specific group for this work [7,12].…”

Search and retrieval of multi-modal data associated with image-parts

“…The main difficulty in cross-media retrieval is to define a similarity measure among heterogeneous low-level features. In order to simultaneously search and retrieve data from multiple modalities, other approaches have been considered [10,11,12,13,14,15,16,17,18,19,20]. For instance in [16], it is experimentally shown that multimodal queries achieve higher retrieval accuracy than mono-modal ones.…”

“…The work in [10] suggests using a combination of ontology browsing and keyword-based querying. The methods presented in [11,14,15,16] use a similar approach and rely on the assumption that every document has an equal number of nearest neighbors for each of the modalities. However, such an assumption might degrade the retrieval performance as a document containing "image+text" may have many nearest neighbors in image modality, but not as many relevant textual data.…”

“…Our approach utilizes the co-occurrence of data points within a modality and across multiple modalities using a graph structure. We note that the proposed work does not impose relevance between each document and an equal number of media objects from all modalities as assumed by [11,14,15,16]. In summary, our contributions are threefold:…”

Multimodal Information Retrieval: Challenges and Future Trends

“…A novel unified framework for multimodal search and retrieval by introducing a novel data representation for multimodal data in Content Object (CO) is described in [12]. Recent efforts in the field of multimodal retrieval systems have led to a growing research community and a number of academic and industrial projects.…”