Flexible Interactive Retrieval SysTem 3.0 for Visual Lifelog Exploration at LSC 2022

Abstract

Building a retrieval system with lifelogging data is more complicated than with ordinary data due to the redundancies, blurriness, massive amount of data, various sources of information accompanying lifelogging data, and especially the ad-hoc nature of queries. The Lifelog Search Challenge (LSC) is a benchmarking challenge that encourages researchers and developers to push the boundaries in lifelog retrieval. For LSC'22, we develop FIRST 3.0, a novel and flexible system that leverages expressive cross-domain embeddings to enhance the searching process. Our system aims to adaptively capture the semantics of an image at different levels of detail. We also propose to augment our system with an external search engine to help our system with initial visual examples for unfamiliar concepts. Finally, we organize image data in hierarchical clusters based on their visual similarity and location to assist users in data exploration. Experiments show that our system is both fast and effective in handling various retrieval scenarios. CCS CONCEPTS• Information systems → Search interfaces; Multimedia databases; • Human-centered computing → Interactive systems and tools.

Evaluating Performance and Trends in Interactive Video Retrieval: Insights From the 12th VBS Competition

“…In VideoCLIP, the integration of Milvus [113] vector database facilitates seamless matching between embeddings. v-FIRST [59] presents a revised version of their previous interactive video retrieval system [110], which supports querying by textual descriptions and visual examples. The joint textvisual feature space is the basis for many of v-FIRST's functionalities, such as optimized vector search, fast neighbor search, and compression of similar video segments.…”

“…v-FIRST indexes different concepts and allows users to apply Boolean retrieval for added flexibility [59]. This combination of concept search and joint embedding is implemented in their unified database.…”

“…This module utilizes feature vectors generated from a fine-tuned GoogleNet architecture [85] and leverages an efficient IVFADC indexing structure [61]. v-FIRST employs optimized nearest neighbor algorithms in the embedding subspace [59] to identify targets that are similar to the example image or text. Furthermore, in v-FIRST, an image generator based on MidJourney and Stable Diffusion is integrated to synthesize images from a text prompt as an additional query methodology.…”

“…v-FIRST implements an optional query reweighting by using the top retrieved images [59]. This approach helps mitigate irrelevant factors and emphasizes concepts that are crucial to the retrieval process.…”

Introduction to the Fifth Annual Lifelog Search Challenge, LSC'22

“…It was improved by the addition of an enhanced text search feature, new filter integration based on the provided metadata, an enhanced CLIP model for additional visual annotation and an enhanced speech-to-query mechanism. FIRST 3.0 [9] participated again with an enhanced system that allows close representation of similar textual/visual data, an extension for external search engine to assist retrieval of unfamiliar concepts, and an option to use query-by-example. It also enhances its user interface to accommodate the new features while maintaining simplicity.…”