Flexible Interactive Retrieval SysTem 2.0 for Visual Lifelog Exploration at LSC 2021

Abstract

With a huge collection of photos and video clips, it is essential to provide an efficient and easy-to-use system for users to retrieve moments of interest with a wide variation of query types. This motivates us to develop and upgrade our flexible interactive retrieval system for visual lifelog exploration. In this paper, we briefly introduce version 2 of our system with the following main features. Our system supports multiple modalities for interaction and query processing, including visual query by meta-data, text query and visual information matching based on a joint embedding model, scene clustering based on visual and location information, flexible temporal event navigation, and query expansion with visual examples. With the flexibility in system architecture, we expect our system can easily integrate new modules to enhance its functionalities. CCS CONCEPTS• Information systems → Search interfaces; Multimedia databases; • Human-centered computing → Interactive systems and tools.

Myscéal: a deeper analysis of an interactive lifelog search engine

“…Another approach is to embed both lifelog images and textual queries into the same vector space where their relevance can be measured by a similarity metric such as cosine distance. Some systems that applied this method have achieved remarkable performance in some lifelog retrieval challenges [12][13][14]. Additionally, knowledge graphs can be employed to extend annotated visual objects to get their synonyms for better retrieval [15] or to capture interaction information in the images [16].…”