An Exploratory Study on a Reinforcement Learning Prototype for Multimodal Image Retrieval Using a Conversational Search Interface

Kaushik, Abhishek; Jacob, Billy; Velavan, Pankaj

doi:10.3390/knowledge2010007

Cited by 5 publications

(2 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, there are other tasks such as image response generation in cases where the system does not have relevant products but wants to generate an image to see if it meets the user's need. Reinforcement learning for Multimodal Image Retrieval could be useful in this case [31]. Moreover, it is more comprehensive to include product images from real-life settings where models actually wear the products in different scenarios such as in the street or doing sports.…”

Section: Discussionmentioning

confidence: 99%

Transformer-Based Multimodal Infusion Dialogue Systems

Liu

et al. 2022

Electronics

View full text Add to dashboard Cite

The recent advancements in multimodal dialogue systems have been gaining importance in several domains such as retail, travel, fashion, among others. Several existing works have improved the understanding and generation of multimodal dialogues. However, there still exists considerable space to improve the quality of output textual responses due to insufficient information infusion between the visual and textual semantics. Moreover, the existing dialogue systems often generate defective knowledge-aware responses for tasks such as providing product attributes and celebrity endorsements. To address the aforementioned issues, we present a Transformer-based Multimodal Infusion Dialogue (TMID) system that extracts the visual and textual information from dialogues via a transformer-based multimodal context encoder and employs a cross-attention mechanism to achieve information infusion between images and texts for each utterance. Furthermore, TMID uses adaptive decoders to generate appropriate multimodal responses based on the user intentions it has determined using a state classifier and enriches the output responses by incorporating domain knowledge into the decoders. The results of extensive experiments on a multimodal dialogue dataset demonstrate that TMID has achieved a state-of-the-art performance by improving the BLUE-4 score by 13.03, NIST by 2.77, image selection Recall@1 by 1.84%.

show abstract

Section: Discussionmentioning

confidence: 99%

Transformer-Based Multimodal Infusion Dialogue Systems

Liu

et al. 2022

Electronics

View full text Add to dashboard Cite

show abstract

“…While this is sufficient for simple question-type queries, it is an inefficient means of engagement for more complex or exploratory queries [19]. In the realm of information, conversational search is a relatively new trend [20]. Conversation is the natural mode for information exchange in daily life [18], and conversational approaches to information retrieval are gaining attention [15].…”

Section: Related Workmentioning

confidence: 99%

A Multi-Modal Foundation Model to Assist People with Blindness and Low Vision in Environmental Interaction

Hao,

Yang,

Huang

et al. 2024

J. Imaging

View full text Add to dashboard Cite

People with blindness and low vision (pBLV) encounter substantial challenges when it comes to comprehensive scene recognition and precise object identification in unfamiliar environments. Additionally, due to the vision loss, pBLV have difficulty in accessing and identifying potential tripping hazards independently. Previous assistive technologies for the visually impaired often struggle in real-world scenarios due to the need for constant training and lack of robustness, which limits their effectiveness, especially in dynamic and unfamiliar environments, where accurate and efficient perception is crucial. Therefore, we frame our research question in this paper as: How can we assist pBLV in recognizing scenes, identifying objects, and detecting potential tripping hazards in unfamiliar environments, where existing assistive technologies often falter due to their lack of robustness? We hypothesize that by leveraging large pretrained foundation models and prompt engineering, we can create a system that effectively addresses the challenges faced by pBLV in unfamiliar environments. Motivated by the prevalence of large pretrained foundation models, particularly in assistive robotics applications, due to their accurate perception and robust contextual understanding in real-world scenarios induced by extensive pretraining, we present a pioneering approach that leverages foundation models to enhance visual perception for pBLV, offering detailed and comprehensive descriptions of the surrounding environment and providing warnings about potential risks. Specifically, our method begins by leveraging a large-image tagging model (i.e., Recognize Anything Model (RAM)) to identify all common objects present in the captured images. The recognition results and user query are then integrated into a prompt, tailored specifically for pBLV, using prompt engineering. By combining the prompt and input image, a vision-language foundation model (i.e., InstructBLIP) generates detailed and comprehensive descriptions of the environment and identifies potential risks in the environment by analyzing environmental objects and scenic landmarks, relevant to the prompt. We evaluate our approach through experiments conducted on both indoor and outdoor datasets. Our results demonstrate that our method can recognize objects accurately and provide insightful descriptions and analysis of the environment for pBLV.

show abstract

Conversational Search for Multimedia Archives

Potyagalova

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

An Exploratory Study on a Reinforcement Learning Prototype for Multimodal Image Retrieval Using a Conversational Search Interface

Cited by 5 publications

References 54 publications

Transformer-Based Multimodal Infusion Dialogue Systems

Transformer-Based Multimodal Infusion Dialogue Systems

A Multi-Modal Foundation Model to Assist People with Blindness and Low Vision in Environmental Interaction

Conversational Search for Multimedia Archives

Contact Info

Product

Resources

About