FIRST - Flexible Interactive Retrieval SysTem for Visual Lifelog Exploration at LSC 2020

Abstract

Lifelog can provide useful insights of our daily activities. It is essential to provide a flexible way for users to retrieve certain events or moments of interest, corresponding to a wide variation of query types. This motivates us to develop FIRST, a Flexible Interactive Retrieval SysTem, to help users to combine or integrate various query components in a flexible manner to handle different query scenarios, such as visual clustering data based on color histogram, visual similarity, GPS location, or scene attributes. We also employ personalized concept detection and image captioning to enhance image understanding from visual lifelog data, and develop an autoencoderlike approach for query text and image feature mapping. Furthermore, we refine the user interface of the retrieval system to better assist users in query expansion and verifying sequential events in a flexible temporal resolution to control the navigation speed through sequences of images. CCS CONCEPTS • Information systems → Search interfaces; Multimedia databases; • Human-centered computing → Interactive systems and tools.

Flexible Interactive Retrieval SysTem 2.0 for Visual Lifelog Exploration at LSC 2021

“…Therefore, this retrieval tool is a successful system for known-item and ad-hoc-search tasks. Both LifeSeeker 2.0 [18] and FIRST v1 [30] also support this approach. LifeSeeker 2.0 focuses on text query using a Bag-of-Words approach with visual concept augmentation [18], while our previous systems [16,30] exploit personalized concepts.…”

“…Figure 1 shows the overview structure of our proposed system. We use the user-interface and system integration platforms from our FIRST version 1 [30] to manage different layouts and user interaction modalities, and query processing components, respectively. We provide five main query processing components in our current system.…”

Graph-Based Indexing and Retrieval of Lifelog Data

“…Generally, all the above systems do not focus on the interaction between objects in the lifelog data or the query. Some systems did make progress by encoding the entire textual input or generating captions for lifelogging images to describe activities appearing in them [30,31]. However, these ideas did not focus on the association between objects in lifelog images.…”

Myscéal 2.0: A Revised Experimental Interactive Lifelog Retrieval System for LSC'21

“…Another common approach is to transform images metadata and queries into the same space through an embedding algorithm to be able to make the comparison. This idea was adopted by almost the half of the participants in the LSC'20 [17,20,26,27,39]. On the other hand, SOMHunter [27] which was the runner-up system of the competition and Exquisitor [17] heavily relied on user's relevance feedback, while LifeGraph [33] explored the potential of indexing the lifelog into a graph structure [30].…”