Memento: A Prototype Lifelog Search Engine for LSC'21

Abstract

In this paper, we introduce a new lifelog retrieval system called Memento that leverages semantic representations of images and textual queries projected into a common latent space to facilitate effective retrieval. It bridges the semantic gap between complex visual scenes/events and user information needs expressed as textual and faceted queries. The system, developed for the 2021 Lifelog Search Challenge, also has a minimalist user interface that includes primary search, temporal search, and visual data filtering components. CCS CONCEPTS• Information systems → Information retrieval; Search interfaces; Retrieval models and ranking.

MEMORIA: A Memory Enhancement and MOment RetrIeval Application for LSC 2022

“…Also using VR interface, the ViRMA prototype system introduced an effective representation and exploration of lifelog data. The Memento system [1], explored semantic representations of images and textual queries with a minimalist user interface. The Pho-toCube [33] interactive retrieval system utilised multidimensional space to explore lifelog data.…”

“…Although in a development stage, this research area has the potential to transform our society. One such challenge is the Lifelog Search Challenge (LSC) workshop 1 , which had the participation of several teams that present their systems in an attempt to solve such problems.…”

Introduction to the Fifth Annual Lifelog Search Challenge, LSC'22

“…Since LSC is a multimodal search challenge, the images were provided with metadata including time stamps, music listening history, biometrics, and location information [6,7], all aligned to UTC time. Associated with each image was a list of visual concepts extracted using the Microsoft Computer Vision API 1 and OCR text that was extracted from the images. Prior to release, all faces and screens were redacted from the images in a fully-automated process.…”

“…Nine teams (with interactive lifelog search engines) participated in the LSC'22 challenge, with eight of the teams having participated at least once previously [6,7]. At LSC'21, the CLIP model had been shown to provide effective retrieval (e.g SOMHunter [11] and Memento [1]), so for LSC'22, several systems integrated the CLIP model (developed by Open AI).…”

Memento 2.0: An Improved Lifelog Search Engine for LSC'22

“…FIRST 2.0 [25] tried to bridge the semantic gap between search queries and images by projecting them into a joint embedding space using a self-attention based model. Memento [1] leveraged imagetext embeddings from the CLIP model to reduce the semantic gap between query and images. Voxento [2] used the same backend as Memento but employed an interactive voice enabled user interface.…”