Image description through fusion based recurrent multi-modal learning

Oruganti, Ram Manohar; Sah, Shagan; Pillai, Suhas; Ptucha, Raymond

doi:10.1109/icip.2016.7533033

Cited by 11 publications

(7 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, in terms of the attention modules in image captioning, there are also all kinds of variants having been explored. For examples, we could easily see the multihead attention in [26], the gate-controlled attention [21], the fully attentive paradigms [28,34], the meshed-connection attention [31], the dual attention [35], etc.…”

Section: Related Workmentioning

confidence: 99%

“…For example, we could easily see the huge improvements by early work of deep neural models in [13,14,15]. Besides, more neural network based models, such as the multimodal learning [16,17], the encoder−decoder framework [18], the attention mechanism [19,20], the compositional architectures [21], the describing method of novel objects [22] and the deep bifurcation network [23], have been proposed one after another.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Geometry Attention Transformer with Position-aware LSTMs for Image Captioning

Wang¹,

Shen²,

Ji³

2021

Preprint

View full text Add to dashboard Cite

In recent years, transformer structures have been widely applied in image captioning with impressive performance. For good captioning results, the geometry and position relations of different visual objects are often thought of as crucial information. Aiming to further promote image captioning by transformers, this paper proposes an improved Geometry Attention Transformer (GAT) model. In order to obtain geometric representation ability, two novel geometry-aware architectures are designed respectively for the encoder and decoder in our GAT. Besides, this model includes the two work modules: i) a geometry gate-controlled self-attention refiner, for explicitly incorporating relative spatial information into image region representations in encoding steps, and ii) a group of position-LSTMs, for precisely informing the decoder of relative word position in generating caption texts. The experiment comparisons on the datasets MS COCO and Flickr30K show that our GAT is efficient, and it could often outperform current stateof-the-art image captioning models.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Geometry Attention Transformer with Position-aware LSTMs for Image Captioning

Wang¹,

Shen²,

Ji³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In automotive or indoor robotic visual perception problems, simple concatenation techniques perform well but they fall short in some applications like video captioning [10,33] or summarization [42] where long term dependencies are required. LSTMs in such cases offer a better alternative [59,45].…”

Section: Feature Aggregationmentioning

confidence: 99%

MultiNet++: Multi-Stream Feature Aggregation and Geometric Loss Strategy for Multi-Task Learning

Chennupati

Sistu

Yogamani

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

View full text Add to dashboard Cite

Multi-task learning is commonly used in autonomous driving for solving various visual perception tasks. It offers significant benefits in terms of both performance and computational complexity. Current work on multi-task learning networks focus on processing a single input image and there is no known implementation of multi-task learning handling a sequence of images. In this work, we propose a multistream multi-task network to take advantage of using feature representations from preceding frames in a video sequence for joint learning of segmentation, depth, and motion. The weights of the current and previous encoder are shared so that features computed in the previous frame can be leveraged without additional computation. In addition, we propose to use the geometric mean of task losses as a better alternative to the weighted average of task losses. The proposed loss function facilitates better handling of the difference in convergence rates of different tasks. Experimental results on KITTI, Cityscapes and SYNTHIA datasets demonstrate that the proposed strategies outperform various existing multi-task learning solutions.

show abstract

“…Yao et al (2016) parinject the image whilst pre-injecting image attributes (or vice versa); and Liu et al (2016) par-inject attributes from the image whilst init-injecting the image vector. Other, less common instantiations include par-injecting the image, but only with the first word (this is not pre-inject as the image is not injected on a separate time step) (Karpathy andFei-Fei, 2015, Hessel et al, 2015); and passing the words through a separate RNN, such that the resulting hidden state vectors are what is combined with the image vector (Oruganti et al, 2016). Many times this architecture is used in order to pass a different representation of the same image with every word so that visual information changes for different parts of the sentence being generated.…”

Section: Types Of Architecturesmentioning

confidence: 99%

“…For example, Zhou et al (2016) perform element-wise multiplication of the image vector with the last generated word’s embedding vector in order to attend to different parts of the image vector. Oruganti et al (2016) pass the image through its own RNN for as many times as there are words in order to use a different image vector for every word. Chen and Zitnick (2014, 2015) use a simple RNN to try to predict what the image vector looks like given a prefix.…”

Section: Introductionmentioning

confidence: 99%

Where to put the image in an image caption generator

Tanti¹,

Gatt²,

Camilleri³

2018

Nat. Lang. Eng.

View full text Add to dashboard Cite

When a recurrent neural network language model is used for caption generation, the image information can be fed to the neural network either by directly incorporating it in the RNN -conditioning the language model by 'injecting' image features -or in a layer following the RNNconditioning the language model by 'merging' image features. While both options are attested in the literature, there is as yet no systematic comparison between the two. In this paper we empirically show that it is not especially detrimental to performance whether one architecture is used or another. The merge architecture does have practical advantages, as conditioning by merging allows the RNN's hidden state vector to shrink in size by up to four times. Our results suggest that the visual and linguistic modalities for caption generation need not be jointly encoded by the RNN as that yields large, memory-intensive models with few tangible advantages in performance; rather, the multimodal integration should be delayed to a subsequent stage.

show abstract

Image description through fusion based recurrent multi-modal learning

Cited by 11 publications

References 9 publications

Geometry Attention Transformer with Position-aware LSTMs for Image Captioning

Geometry Attention Transformer with Position-aware LSTMs for Image Captioning

MultiNet++: Multi-Stream Feature Aggregation and Geometric Loss Strategy for Multi-Task Learning

Where to put the image in an image caption generator

Contact Info

Product

Resources

About