vitrivr at the Lifelog Search Challenge 2022

Heller, Silvan; Rossetto, Luca; Sauter, Loris; Schuldt, Heiko

doi:10.1145/3512729.3533003

Cited by 13 publications

(5 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The most common approach to organizing, retrieving, and analyzing data from wearable cameras involves assigning semantic contexts to images, like visual descriptions, time, and location [ 13 , 14 ]. Various computer vision models are employed to extract visual information from the images, including object detection, activity recognition, optical character recognition [ 13 , 15 ], and embedding models [ 16 , 17 ]. A typical retrieval system would also incorporate different techniques, namely, query enhancement [ 13 ], visual similarity search [ 16 ], and temporal search [ 16 ].…”

Section: Discussionmentioning

confidence: 99%

“…Various computer vision models are employed to extract visual information from the images, including object detection, activity recognition, optical character recognition [ 13 , 15 ], and embedding models [ 16 , 17 ]. A typical retrieval system would also incorporate different techniques, namely, query enhancement [ 13 ], visual similarity search [ 16 ], and temporal search [ 16 ]. We previously reviewed the SIB image data manually to determine the feasibility and acceptability of wearable cameras to assess self-care in people living with HF [ 21 ].…”

Section: Discussionmentioning

confidence: 99%

“…Various computer vision models are employed, such as object detection, activity recognition, and optical character recognition, in addition to embedding models [ 13 , 15 - 17 ]. Retrieval systems incorporate techniques such as query enhancement, visual similarity search, and temporal search [ 13 , 16 ]. Previous studies [ 18 - 20 ], including our own [ 21 ], have manually reviewed camera images, which is time-consuming.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Daily Activity Lifelogs of People With Heart Failure: Observational Study

Tegegne,

Tran,

Nourse

et al. 2024

JMIR Form Res

View full text Add to dashboard Cite

Background Globally, heart failure (HF) affects more than 64 million people, and attempts to reduce its social and economic burden are a public health priority. Interventions to support people with HF to self-manage have been shown to reduce hospitalizations, improve quality of life, and reduce mortality rates. Understanding how people self-manage is imperative to improve future interventions; however, most approaches to date, have used self-report methods to achieve this. Wearable cameras provide a unique tool to understand the lived experiences of people with HF and the daily activities they undertake, which could lead to more effective interventions. However, their potential for understanding chronic conditions such as HF is unclear. Objective This study aimed to determine the potential utility of wearable cameras to better understand the activities of daily living in people living with HF. Methods The “Seeing is Believing (SIB)” study involved 30 patients with HF who wore wearable cameras for a maximum of 30 days. We used the E-Myscéal web-based lifelog retrieval system to process and analyze the wearable camera image data set. Search terms for 7 daily activities (physical activity, gardening, shopping, screen time, drinking, eating, and medication intake) were developed and used for image retrieval. Sensitivity analysis was conducted to compare the number of images retrieved using different search terms. Temporal patterns in daily activities were examined, and differences before and after hospitalization were assessed. Results E-Myscéal exhibited sensitivity to specific search terms, leading to significant variations in the number of images retrieved for each activity. The highest number of images returned were related to eating and drinking, with fewer images for physical activity, screen time, and taking medication. The majority of captured activities occurred before midday. Notably, temporal differences in daily activity patterns were observed for participants hospitalized during this study. The number of medication images increased after hospital discharge, while screen time images decreased. Conclusions Wearable cameras offer valuable insights into daily activities and self-management in people living with HF. E-Myscéal efficiently retrieves relevant images, but search term sensitivity underscores the need for careful selection.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Daily Activity Lifelogs of People With Heart Failure: Observational Study

Tegegne,

Tran,

Nourse

et al. 2024

JMIR Form Res

View full text Add to dashboard Cite

show abstract

“…This makes the model more robust against various nuances or typos in the text query and allows users to construct more natural descriptions of the searched image, rather than "keyword-style" descriptions. The impressive performance of CLIP has led to its adoption by teams in the following year [49].…”

Section: B Multimodal Embeddingsmentioning

confidence: 99%

Comparing Interactive Retrieval Approaches at the Lifelog Search Challenge 2021

et al. 2023

View full text Add to dashboard Cite

The Lifelog Search Challenge (LSC) is an interactive benchmarking evaluation workshop for lifelog retrieval systems. The challenge was first organised in 2018 aiming to find the system that can quickly retrieve relevant lifelog images for a given semantic query. This paper provides an analysis of the performance of all 17 systems participating in the 4th LSC workshop held at the 2021 Annual ACM International Conference on Multimedia Retrieval (ICMR). LSC'21 was the largest effort at comparing different approaches to interactive lifelog retrieval systems seen thus far. Findings from the challenge suggest that many different interactive factors contribute to the success (or otherwise) of participating teams. In this paper, we provide an overview of the LSC'21 challenge, introduce each team's approach and explore these factors in depth and offer clues on how to develop a high-performing interactive lifelog search engine. INDEX TERMS lifelog, information retrieval, multimodal, analyticsAt LSC'21, each of the participating teams brought a unique and customised search engine to the challenge. In this paper, we introduce the LSC challenge, describe all competing systems, and highlight the techniques and components that are employed in state-of-the-art interactive lifelog

show abstract

“…Enhancements focused on integrating the CLIP model for retrieval and providing a novice user-friendly search interface. The vitrivr system, a general purpose multimedia data indexing and retrieval framework [8], supported lifelog search using a number of fundamental techniques, such as text embedding, visual similarity search, classical Boolean operators as well as supporting lifelog-specific enhancements such as dynamic result sequencing and faceted filtering. Built on-top of the vitrivr stack was vitrivr-VR [15], which provided a VR-based access mechanism to the lifelog dataset.…”

Section: Participating Systemsmentioning

confidence: 99%

Introduction to the Fifth Annual Lifelog Search Challenge, LSC'22

Gurrin

Zhou

Healy

et al. 2022

Proceedings of the 2022 International Conference on Multimedia Retrieval

Self Cite

View full text Add to dashboard Cite

For the fifth time since 2018, the Lifelog Search Challenge (LSC) facilitated a benchmarking exercise to compare interactive search systems designed for multimodal lifelogs. LSC'22 attracted nine participating research groups who developed interactive lifelog retrieval systems enabling fast and effective access to lifelogs. The systems competed in front of a hybrid audience at the LSC workshop at ACM ICMR'22. This paper presents an introduction to the LSC workshop, the new (larger) dataset used in the competition, and introduces the participating lifelog search systems.

show abstract

vitrivr at the Lifelog Search Challenge 2022

Cited by 13 publications

References 37 publications

Daily Activity Lifelogs of People With Heart Failure: Observational Study

Daily Activity Lifelogs of People With Heart Failure: Observational Study

Comparing Interactive Retrieval Approaches at the Lifelog Search Challenge 2021

Introduction to the Fifth Annual Lifelog Search Challenge, LSC'22

Contact Info

Product

Resources

About