Memento 2.0: An Improved Lifelog Search Engine for LSC'22

Alam, Naushad; Graham, Yvette; Gurrin, Cathal

doi:10.1145/3512729.3533006

Cited by 16 publications

(7 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The most common approach to organizing, retrieving, and analyzing data from wearable cameras involves assigning semantic contexts to images, like visual descriptions, time, and location [ 13 , 14 ]. Various computer vision models are employed to extract visual information from the images, including object detection, activity recognition, optical character recognition [ 13 , 15 ], and embedding models [ 16 , 17 ]. A typical retrieval system would also incorporate different techniques, namely, query enhancement [ 13 ], visual similarity search [ 16 ], and temporal search [ 16 ].…”

Section: Discussionmentioning

confidence: 99%

“…This involves assigning semantic contexts like visual descriptions, time, and location [ 13 , 14 ]. Various computer vision models are employed, such as object detection, activity recognition, and optical character recognition, in addition to embedding models [ 13 , 15 - 17 ]. Retrieval systems incorporate techniques such as query enhancement, visual similarity search, and temporal search [ 13 , 16 ].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Daily Activity Lifelogs of People With Heart Failure: Observational Study

Tegegne,

Tran,

Nourse

et al. 2024

JMIR Form Res

Self Cite

View full text Add to dashboard Cite

Background Globally, heart failure (HF) affects more than 64 million people, and attempts to reduce its social and economic burden are a public health priority. Interventions to support people with HF to self-manage have been shown to reduce hospitalizations, improve quality of life, and reduce mortality rates. Understanding how people self-manage is imperative to improve future interventions; however, most approaches to date, have used self-report methods to achieve this. Wearable cameras provide a unique tool to understand the lived experiences of people with HF and the daily activities they undertake, which could lead to more effective interventions. However, their potential for understanding chronic conditions such as HF is unclear. Objective This study aimed to determine the potential utility of wearable cameras to better understand the activities of daily living in people living with HF. Methods The “Seeing is Believing (SIB)” study involved 30 patients with HF who wore wearable cameras for a maximum of 30 days. We used the E-Myscéal web-based lifelog retrieval system to process and analyze the wearable camera image data set. Search terms for 7 daily activities (physical activity, gardening, shopping, screen time, drinking, eating, and medication intake) were developed and used for image retrieval. Sensitivity analysis was conducted to compare the number of images retrieved using different search terms. Temporal patterns in daily activities were examined, and differences before and after hospitalization were assessed. Results E-Myscéal exhibited sensitivity to specific search terms, leading to significant variations in the number of images retrieved for each activity. The highest number of images returned were related to eating and drinking, with fewer images for physical activity, screen time, and taking medication. The majority of captured activities occurred before midday. Notably, temporal differences in daily activity patterns were observed for participants hospitalized during this study. The number of medication images increased after hospital discharge, while screen time images decreased. Conclusions Wearable cameras offer valuable insights into daily activities and self-management in people living with HF. E-Myscéal efficiently retrieves relevant images, but search term sensitivity underscores the need for careful selection.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Daily Activity Lifelogs of People With Heart Failure: Observational Study

Tegegne,

Tran,

Nourse

et al. 2024

JMIR Form Res

Self Cite

View full text Add to dashboard Cite

show abstract

“…Furthermore, this search engine could only capture the objects but not other information, such as the interaction between objects, the visible text in images, or the context of the images. Recently, many current lifelog retrieval systems [31,2,23] used CLIP to deal with those problems and obtained the top performance at LSC '22 [12]. This was because the CLIP model learns not only the general information of images but also detailed information, such as the visible text in them.…”

Section: Related Workmentioning

confidence: 99%

“…CLIP in Lifelog Retrieval. Recent works [2,27] experimented to show the performance of different versions of CLIP in the lifelog retrieval task. However, unlike their experiments, we compared the concept-based model with many SOTA crossmodality retrieval models, including CLIP, BLIP, and HADA, in the lifelog retrieval task with two conőgurations: automatic manner and interactive manner.…”

Section: Related Workmentioning

confidence: 99%

Concept-Based and Embedding-Based Models in Lifelog Retrieval: An Empirical Comparison of Performance

Nguyen,

Gurrin

2024

Preprint

View full text Add to dashboard Cite

Many lifelog retrieval systems have been introduced that apply various approaches to their search engines. The traditional method was to match concepts, which are visual objects detected in images and semantic queries. This concept-based approach has been applied in many retrieval systems, achieving the top performance in lifelog search challenges. Many novel embedding-based cross-modality retrieval models, such as CLIP, BLIP, or HADA, have been developed recently and obtained state-of-the-art (SOTA) results in the image-text retrieval task. These models have recently been applied in several lifelog search challenges. However, there is no comprehensive comparison between them since many benchmarking evaluations contain bias factors such as different user interfaces of participated lifelog retrieval systems. In this paper, we conducted non-biased experiments in both automatic (non-interactive) and interactive configurations to evaluate the performance of many SOTA retrieval models, including the traditional concept-based approach, in the lifelog retrieval task. Furthermore, we retrained the models in a lifelog Q&A dataset to assess whether retraining on a small lifelog dataset could improve the performance. The result showed that embedding-based search engines outperformed the concept-based approach by a large margin in both settings. The finding opens the opportunity to apply the embedding-based models as a new generation of lifelog retrieval models instead of the conventional concept-based approach. The source code will be available soon for reproducibility.

show abstract

“…It also enhances its user interface to accommodate the new features while maintaining simplicity. Memento 2.0 [2] utilised a weighted ensemble approach to CLIP integration, which significantly improved the performance over the LSC'21 system and it also introduced a number of updates to the UI to enhance user efficiency.…”

Section: Participating Systemsmentioning

confidence: 99%

Introduction to the Fifth Annual Lifelog Search Challenge, LSC'22

Gurrin

Zhou

Healy

et al. 2022

Proceedings of the 2022 International Conference on Multimedia Retrieval

Self Cite

View full text Add to dashboard Cite

For the fifth time since 2018, the Lifelog Search Challenge (LSC) facilitated a benchmarking exercise to compare interactive search systems designed for multimodal lifelogs. LSC'22 attracted nine participating research groups who developed interactive lifelog retrieval systems enabling fast and effective access to lifelogs. The systems competed in front of a hybrid audience at the LSC workshop at ACM ICMR'22. This paper presents an introduction to the LSC workshop, the new (larger) dataset used in the competition, and introduces the participating lifelog search systems.

show abstract

Memento 2.0: An Improved Lifelog Search Engine for LSC'22

Cited by 16 publications

References 26 publications

Daily Activity Lifelogs of People With Heart Failure: Observational Study

Daily Activity Lifelogs of People With Heart Failure: Observational Study

Concept-Based and Embedding-Based Models in Lifelog Retrieval: An Empirical Comparison of Performance

Introduction to the Fifth Annual Lifelog Search Challenge, LSC'22

Contact Info

Product

Resources

About