Multi-step Joint-Modality Attention Network for Scene-Aware Dialogue System

Chu, Yun-Wei; Lin, Kuan-Yen; Hsu, Chao-Chun; Ku, Lun-Wei

doi:10.48550/arxiv.2001.06206

Cited by 8 publications

(4 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Baselines. To demonstrate the effectiveness of our proposed model, we compare with several baseline methods: (i) Baseline [5], (ii) JMAN [19], an multi-step joint-modality attention network, (iii) RLM [6], which uses pre-trained language model to process multimodal inputs, (iv) SCGA [20], using co-reference graph attention to deduce correlation among multimodalities, and (v) PDC [21], that uses both semantic graph and pre-trained language model. For a thorough comparison, we also implement another variant of our model: BART (I3D+VGGish), which directly feeds the offline-extracted video features into BART as video embeddings.…”

Section: Resultsmentioning

confidence: 99%

Video-Grounded Dialogues with Joint Video and Image Training

Zhang

2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

In this paper, we propose a multi-modal transformer model for end-to-end training of video-grounded dialogue generation. In particular, LayerScale regularized spatio-temporal self-attention blocks are first introduced to enable us to flexibly train end-to-end from both video and image data, without extracting offline visual features. Further, a pre-trained generative language architecture BART is employed to encode different modalities and perform dialogue generation. Extensive experiments on Audio-Visual Scene-Aware Dialog (AVSD) dataset demonstrate its effectiveness and superiority to the state-of-the-art methods.

show abstract

Section: Resultsmentioning

confidence: 99%

Video-Grounded Dialogues with Joint Video and Image Training

Zhang

2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

show abstract

“…Li et al [ 22 ] proposed a transformer-based generative framework that integrates all the modalities by encoding features into the system and generates better multimodal-based system responses using multi-task learning methods. Chu et al [ 38 ] described a consecutive multimodal fusion strategy using joint modal attention during conversation. Although these approaches exhibit significant performances, they have two limitations.…”

Section: Related Workmentioning

confidence: 99%

Natural-Language-Driven Multimodal Representation Learning for Audio-Visual Scene-Aware Dialog System

Heo,

Kang,

Seo

2023

Sensors

View full text Add to dashboard Cite

With the development of multimedia systems in wireless environments, the rising need for artificial intelligence is to design a system that can properly communicate with humans with a comprehensive understanding of various types of information in a human-like manner. Therefore, this paper addresses an audio-visual scene-aware dialog system that can communicate with users about audio-visual scenes. It is essential to understand not only visual and textual information but also audio information in a comprehensive way. Despite the substantial progress in multimodal representation learning with language and visual modalities, there are still two caveats: ineffective use of auditory information and the lack of interpretability of the deep learning systems’ reasoning. To address these issues, we propose a novel audio-visual scene-aware dialog system that utilizes a set of explicit information from each modality as a form of natural language, which can be fused into a language model in a natural way. It leverages a transformer-based decoder to generate a coherent and correct response based on multimodal knowledge in a multitask learning setting. In addition, we also address the way of interpreting the model with a response-driven temporal moment localization method to verify how the system generates the response. The system itself provides the user with the evidence referred to in the system response process as a form of the timestamp of the scene. We show the superiority of the proposed model in all quantitative and qualitative measurements compared to the baseline. In particular, the proposed model achieved robust performance even in environments using all three modalities, including audio. We also conducted extensive experiments to investigate the proposed model. In addition, we obtained state-of-the-art performance in the system response reasoning task.

show abstract

“…Early attempts [14,24,33,35] employ the recurrent neural network to encode dialog history. Later methods [6,30,49] use the Attention mechanism, or [45] design a memory networks to extract the relationship between different modalities, [22,29,26] employ Transformer-based network to resolve the cross modality learning. More explicit relationships in dialogs are recently studied, showing promising results [12,23].…”

Section: Related Workmentioning

confidence: 99%

Video Dialog as Conversation about Objects Living in Space-Time

Pham¹,

Le²,

Le³

et al. 2022

Preprint

View full text Add to dashboard Cite

It would be a technological feat to be able to create a system that can hold a meaningful conversation with humans about what they watch. A setup toward that goal is presented as a video dialog task, where the system is asked to generate natural utterances in response to a question in an ongoing dialog. The task poses great visual, linguistic, and reasoning challenges that cannot be easily overcome without an appropriate representation scheme over video and dialog that supports high-level reasoning. To tackle these challenges we present a new objectcentric framework for video dialog that supports neural reasoning dubbed COST-which stands for Conversation about Objects in Space-Time. Here dynamic space-time visual content in videos is first parsed into object trajectories. Given this video abstraction, COST maintains and tracks object-associated dialog states, which are updated upon receiving new questions. Object interactions are dynamically and conditionally inferred for each question, and these serve as the basis for relational reasoning among them. COST also maintains a history of previous answers, and this allows retrieval of relevant object-centric information to enrich the answer forming process. Language production then proceeds in a stepwise manner, taking the context of the current utterance, the existing dialog, and the current question. We evaluate COST on the AVSD test splits (DSTC7 and DSTC8), demonstrating its competitiveness against state-of-the-arts.

show abstract

Multi-step Joint-Modality Attention Network for Scene-Aware Dialogue System

Cited by 8 publications

References 17 publications

Video-Grounded Dialogues with Joint Video and Image Training

Video-Grounded Dialogues with Joint Video and Image Training

Natural-Language-Driven Multimodal Representation Learning for Audio-Visual Scene-Aware Dialog System

Video Dialog as Conversation about Objects Living in Space-Time

Contact Info

Product

Resources

About