E-Myscéal: Embedding-based Interactive Lifelog Retrieval System for LSC'22

Tran, Ly-Duyen; Nguyen, Manh-Duy; Nguyen, Binh T.; Lee, Hyowon; Zhou, Liting; Gurrin, Cathal

doi:10.1145/3512729.3533012

Cited by 32 publications

(19 citation statements)

References 31 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The wearable camera image data were processed and retrieved using the E-Myscéal web-based lifelog retrieval system. The E-Myscéal system uses deep learning algorithms to create embeddings (vector representations) of various lifelog data types such as images, text, and audio, enabling intuitive and web-based cross-media querying [ 22 ]. E-Myscéal uses the CLIP model to retrieve images similar in content to descriptive textual search terms, allowing users to search through lifelog data with great flexibility [ 27 ].…”

Section: Methodsmentioning

confidence: 99%

“…With E-Myscéal, users can enter any search terms, leading to a list of related images along with date or time and location metadata [ 22 ]. For instance, when “eating” was used as a search term, E-Myscéal retrieved all food-related camera images, enabling researchers to assess the frequency of the wearer’s food consumption per day or over multiple days (see Figure 1 ).…”

Section: Methodsmentioning

confidence: 99%

“…For instance, when “eating” was used as a search term, E-Myscéal retrieved all food-related camera images, enabling researchers to assess the frequency of the wearer’s food consumption per day or over multiple days (see Figure 1 ). The system’s flexibility allows researchers to query data sets for any desired search terms, making it useful for various applications [ 22 ]. A demonstration of this is evident in the grouping of images within a scene , showcased in the right section of the pop-up within Figure 1 .…”

Section: Methodsmentioning

confidence: 99%

“…Previous studies [ 18 - 20 ], including our own [ 21 ], have manually reviewed camera images, which is time-consuming. In this study, we used the E-Myscéal system [ 22 ] for efficient retrieval and review of relevant images depicting the daily activities of people living with HF.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Daily Activity Lifelogs of People With Heart Failure: Observational Study

Tegegne,

Tran,

Nourse

et al. 2024

JMIR Form Res

Self Cite

View full text Add to dashboard Cite

Background Globally, heart failure (HF) affects more than 64 million people, and attempts to reduce its social and economic burden are a public health priority. Interventions to support people with HF to self-manage have been shown to reduce hospitalizations, improve quality of life, and reduce mortality rates. Understanding how people self-manage is imperative to improve future interventions; however, most approaches to date, have used self-report methods to achieve this. Wearable cameras provide a unique tool to understand the lived experiences of people with HF and the daily activities they undertake, which could lead to more effective interventions. However, their potential for understanding chronic conditions such as HF is unclear. Objective This study aimed to determine the potential utility of wearable cameras to better understand the activities of daily living in people living with HF. Methods The “Seeing is Believing (SIB)” study involved 30 patients with HF who wore wearable cameras for a maximum of 30 days. We used the E-Myscéal web-based lifelog retrieval system to process and analyze the wearable camera image data set. Search terms for 7 daily activities (physical activity, gardening, shopping, screen time, drinking, eating, and medication intake) were developed and used for image retrieval. Sensitivity analysis was conducted to compare the number of images retrieved using different search terms. Temporal patterns in daily activities were examined, and differences before and after hospitalization were assessed. Results E-Myscéal exhibited sensitivity to specific search terms, leading to significant variations in the number of images retrieved for each activity. The highest number of images returned were related to eating and drinking, with fewer images for physical activity, screen time, and taking medication. The majority of captured activities occurred before midday. Notably, temporal differences in daily activity patterns were observed for participants hospitalized during this study. The number of medication images increased after hospital discharge, while screen time images decreased. Conclusions Wearable cameras offer valuable insights into daily activities and self-management in people living with HF. E-Myscéal efficiently retrieves relevant images, but search term sensitivity underscores the need for careful selection.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Daily Activity Lifelogs of People With Heart Failure: Observational Study

Tegegne,

Tran,

Nourse

et al. 2024

JMIR Form Res

Self Cite

View full text Add to dashboard Cite

show abstract

“…To show the stable performance of HADA, we used it to combine 2 other different pretrained models, including BLIP [17] and CLIP [28]. While CLIP is well-known for its application in many retrieval challenges [24,32,9,31], BLIP is the enhanced version of ALBEF with the bootstrapping technique in the training process. We used the same configuration as described in 4.2 to train and evaluate HADA in Flickr30k and MSCOCO datasets.…”

Section: Ablation Studymentioning

confidence: 99%

HADA: A Graph-based Amalgamation Framework in Image-text Retrieval

Nguyen¹,

Nguyen²,

Gurrin³

2023

Preprint

View full text Add to dashboard Cite

Many models have been proposed for vision and language tasks, especially the image-text retrieval task. All state-of-the-art (SOTA) models in this challenge contained hundreds of millions of parameters. They also were pretrained on a large external dataset that has been proven to make a big improvement in overall performance. It is not easy to propose a new model with a novel architecture and intensively train it on a massive dataset with many GPUs to surpass many SOTA models, which are already available to use on the Internet. In this paper, we proposed a compact graph-based framework, named HADA, which can combine pretrained models to produce a better result, rather than building from scratch. First, we created a graph structure in which the nodes were the features extracted from the pretrained models and the edges connecting them. The graph structure was employed to capture and fuse the information from every pretrained model with each other. Then a graph neural network was applied to update the connection between the nodes to get the representative embedding vector for an image and text. Finally, we used the cosine similarity to match images with their relevant texts and vice versa to ensure a low inference time. Our experiments showed that, although HADA contained a tiny number of trainable parameters, it could increase baseline performance by more than 3.6% in terms of evaluation metrics in the Flickr30k dataset. Additionally, the proposed model did not train on any external dataset and did not require many GPUs but only 1 to train due to its small number of parameters. The source code is available at https://github.com/m2man/HADA.

show abstract