Text-Guided Attention Model for Image Captioning

Mun, Jonghwan; Cho, Minsu; Han, Bohyung

doi:10.1609/aaai.v31i1.11237

Cited by 45 publications

(6 citation statements)

References 14 publications

(21 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Text-guided attention approach was presented by Mun et al [105]. Related sample captions, namely guidance captions, were employed to get visual attention and produce appropriate captions.…”

Section: Guided Attentionmentioning

confidence: 99%

A Survey on Attention-Based Models for Image Captioning

Osman¹,

Shalaby²,

Soliman³

et al. 2023

IJACSA

View full text Add to dashboard Cite

Image captioning task is highly used in many realworld applications. The captioning task is concerned with understanding the image using computer vision methods. Then, natural language processing methods are used to produce a description for the image. Different approaches were proposed to solve this task, and deep learning attention-based models have been proven to be the state-of-the-art. A survey on attentionbased models for image captioning is presented in this paper including new categories that were not included in other survey papers. The attention-based approaches are classified into four main categories, further classified into subcategories. All categories and subcategories of the attention-based approaches are discussed in detail. Furthermore, the state-of-the-art approaches are compared and the accuracy improvements are stated especially in the transformer-based models, and a summary of the benchmark datasets and the main performance metrics is presented.

show abstract

“…Text-guided attention approach was presented by Mun et al [105]. Related sample captions, namely guidance captions, were employed to get visual attention and produce appropriate captions.…”

Section: Guided Attentionmentioning

confidence: 99%

A Survey on Attention-Based Models for Image Captioning

Osman¹,

Shalaby²,

Soliman³

et al. 2023

IJACSA

View full text Add to dashboard Cite

show abstract

“…In the VQA task, self-attention is extensively used to model word-to-word relationships for questions and region-to-region relationships for images in the VQA task. Questionguided attention on image regions or video frames is generally explored for visual question answering [27,36,45,46], video question answering [47][48][49], image captioning [6], etc. In order to capture more intensive correlation between cross-modal nodes, co-attention-based approaches [25,29,36,37] use bi-direction attention to learn the relationships between wordregion pairs.…”

Section: Attention Mechanismsmentioning

confidence: 99%

“…At present, multi-modal learning has bridged the gap between visual and language and has been widely concerned [1][2][3][4][5]. Remarkable progress has been made in many multi-modal learning tasks, e.g., image captioning [6][7][8][9], video captioning [10][11][12], cross-modal retrieval [13][14][15][16][17][18][19][20][21][22], and visual question answering(VQA) [7,[23][24][25][26][27][28][29][30][31].…”

Section: Introductionmentioning

confidence: 99%

Path-Wise Attention Memory Network for Visual Question Answering

et al. 2022

View full text Add to dashboard Cite

Visual question answering (VQA) is regarded as a multi-modal fine-grained feature fusion task, which requires the construction of multi-level and omnidirectional relations between nodes. One main solution is the composite attention model which is composed of co-attention (CA) and self-attention(SA). However, the existing composite models only consider the stack of single attention blocks, lack of path-wise historical memory, and overall adjustments. We propose a path attention memory network (PAM) to construct a more robust composite attention model. After each single-hop attention block (SA or CA), the importance of the cumulative nodes is used to calibrate the signal strength of nodes’ features. Four memoried single-hop attention matrices are used to obtain the path-wise co-attention matrix of path-wise attention (PA); therefore, the PA block is capable of synthesizing and strengthening the learning effect on the whole path. Moreover, we use guard gates of the target modal to check the source modal values in CA and conditioning gates of another modal to guide the query and key of the current modal in SA. The proposed PAM is beneficial to construct a robust multi-hop neighborhood relationship between visual and language and achieves excellent performance on both VQA2.0 and VQA-CP V2 datasets.

show abstract

“…State-of-the-art and limitations: The pursuit of multimodalinput-based abstractive text summarization can be related to various other fields, such as image and video captioning [22,34,39,48,49], video story generation [16], video title generation [57], and multimodal sentence summarization [28]. However, these works generally produce summaries based on either images or short videos, and the target summaries are easier to predict due to the limited vocabulary diversity.…”

Section: Introductionmentioning

confidence: 99%

An abstractive text summarization technique using transformer model with self-attention mechanism

Kumar

Solanki

2023

Neural Comput & Applic

View full text Add to dashboard Cite

The realm of scientific text summarization has experienced remarkable progress due to the availability of annotated brief summaries and ample data. However, the utilization of multiple input modalities, such as videos and audio, has yet to be thoroughly explored. At present, scientific multimodal-input-based text summarization systems tend to employ longer target summaries like abstracts, leading to an underwhelming performance in the task of text summarization.In this paper, we deal with a novel task of extreme abstractive text summarization (aka TL;DR generation) by leveraging multiple input modalities. To this end, we introduce mTLDR, a first-of-itskind dataset for the aforementioned task, comprising videos, audio, and text, along with both author-composed summaries and expertannotated summaries. The mTLDR dataset accompanies a total of 4, 182 instances collected from various academic conference proceedings, such as ICLR, ACL, and CVPR. Subsequently, we present mTLDRgen, an encoder-decoder-based model that employs a novel dual-fused hyper-complex Transformer combined with a Wasserstein Riemannian Encoder Transformer, to dexterously capture the intricacies between different modalities in a hyper-complex latent geometric space. The hyper-complex Transformer captures the intrinsic properties between the modalities, while the Wasserstein Riemannian Encoder Transformer captures the latent structure of the modalities in the latent space geometry, thereby enabling the model to produce diverse sentences. mTLDRgen outperforms 20 baselines on mTLDR as well as another non-scientific dataset (How2) across three Rouge-based evaluation measures. Furthermore, based on the qualitative metrics, BERTScore and FEQA, and human evaluations, we demonstrate that the summaries generated by mTLDRgen are fluent and congruent to the original source material. CCS CONCEPTS• >Computing methodologies → Natural language processing.

show abstract

Text-Guided Attention Model for Image Captioning

Cited by 45 publications

References 14 publications

A Survey on Attention-Based Models for Image Captioning

A Survey on Attention-Based Models for Image Captioning

Path-Wise Attention Memory Network for Visual Question Answering

An abstractive text summarization technique using transformer model with self-attention mechanism

Contact Info

Product

Resources

About