Graph-Based Indexing and Retrieval of Lifelog Data

Frontiers in Artificial Intelligence and Applications

Gurrin

2021

Self Cite

Conventional approaches to image-text retrieval mainly focus on indexing visual objects appearing in pictures but ignore the interactions between these objects. Such objects occurrences and interactions are equivalently useful and important in this field as they are usually mentioned in the text. Scene graph presentation is a suitable method for the image-text matching challenge and obtained good results due to its ability to capture the inter-relationship information. Both images and text are represented in scene graph levels and formulate the retrieval challenge as a scene graph matching challenge. In this paper, we introduce the Local and Global Scene Graph Matching (LGSGM) model that enhances the state-of-the-art method by integrating an extra graph convolution network to capture the general information of a graph. Specifically, for a pair of scene graphs of an image and its caption, two separate models are used to learn the features of each graph’s nodes and edges. Then a Siamese-structure graph convolution model is employed to embed graphs into vector forms. We finally combine the graph-level and the vector-level to calculate the similarity of this image-text pair. The empirical experiments show that our enhancement with the combination of levels can improve the performance of the baseline method by increasing the recall by more than 10% on the Flickr30k dataset. Our implementation code can be found at https://github.com/m2man/LGSGM.

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A Deep Local and Global Scene-Graph Matching for Image-Text Retrieval

Frontiers in Artificial Intelligence and Applications

Gurrin

2021

Self Cite

“…Another VR-based system, called VIRMA [7], created a hierarchy structure of visual concepts that supports users to explore the lifelog data rather than retrieval. Some systems applied the graph structure to lifelog dataset [24,27]. Lifegraph [28] used the knowledge graph to enhance the annotated information.…”

Section: Related Workmentioning

confidence: 99%

E-Myscéal: Embedding-based Interactive Lifelog Retrieval System for LSC'22

Tran

Proceedings of the 5th Annual on Lifelog Search Challenge

et al. 2022

Self Cite

Developing interactive lifelog retrieval systems is a growing research area. There are many international competitions for lifelog retrieval that encourage researchers to build effective systems that can address the multimodal retrieval challenge of lifelogs. The Lifelog Search Challenge (LSC) was first organised in 2018 and is currently the only interactive benchmarking evaluation for lifelog retrieval systems. Participating systems should have an accurate search engine and a user-friendly interface that can help users to retrieve relevant content. In this paper, we upgrade our previous Myscéal, which was the top performing system in LSC'20 and LSC'21, and present E-Myscéal for LSC'22, which includes a completely different search engine. Instead of using visual concepts for retrieval such as Myscéal, the new E-Myscéal employs an embedding technique that facilitates novice users who are not familiar with the concepts. Our experiments show that the new search engine can find relevant images in the first place in the ranked list, four a quarter of the LSC'21 queries (26%) by using just the first hint from the textual information need. Regarding the user interface, we still keep the simple non-faceted design as in the previous version but improve the event view browsing in order to better support novice users. CCS CONCEPTS• Information systems → Information retrieval; • Humancentered computing → Human computer interaction (HCI); User interface design.

“…This idea was adopted by almost the half of the participants in the LSC'20 [17,20,26,27,39]. On the other hand, SOMHunter [27] which was the runner-up system of the competition and Exquisitor [17] heavily relied on user's relevance feedback, while LifeGraph [33] explored the potential of indexing the lifelog into a graph structure [30].…”

Section: Related Workmentioning

confidence: 99%

Myscéal 2.0: A Revised Experimental Interactive Lifelog Retrieval System for LSC'21

Tran

Proceedings of the 4th Annual on Lifelog Search Challenge

Bình

et al. 2021

Self Cite

Building an interactive retrieval system for lifelogging contains many challenges due to massive multi-modal personal data besides the requirement of accuracy and rapid response for such a tool. The Lifelog Search Challenge (LSC) is the international lifelog retrieval competition that inspires researchers to develop their systems to cope with the challenges and evaluates the effectiveness of their solutions. In this paper, we upgrade our previous Myscéal and present Myscéal 2.0 system for the LSC'21 with the improved features inspired by the novice users experiments. The experiments show that a novice user achieved more than half of the expert score on average. To mitigate the gap of them, some potential enhancements were identified and integrated to the enhanced version. CCS CONCEPTS• Information systems → Information retrieval; • Humancentered computing → Human computer interaction (HCI); User interface design.