Recall What You See Continually Using GridLSTM in Image Captioning

Wu, Lingxiang; Xu, Min; Wang, Jinqiao; Perry, Stuart

doi:10.1109/tmm.2019.2931815

Cited by 35 publications

(6 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Human beings have the potential to extract visual information from images [1][2][3]. e main objective is to utilize this human ability to generate meaning-full textual information from digital images to design automatic medical image captioning [4,5].…”

Section: Introductionmentioning

confidence: 99%

Medical Image Captioning Using Optimized Deep Learning Model

Singh

Raguru

Prasad

et al. 2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

Medical image captioning provides the visual information of medical images in the form of natural language. It requires an efficient approach to understand and evaluate the similarity between visual and textual elements and to generate a sequence of output words. A novel show, attend, and tell model (ATM) is implemented, which considers a visual attention approach using an encoder-decoder model. But the show, attend, and tell model is sensitive to its initial parameters. Therefore, a Strength Pareto Evolutionary Algorithm-II (SPEA-II) is utilized to optimize the initial parameters of the ATM. Finally, experiments are considered using the benchmark data sets and competitive medical image captioning techniques. Performance analysis shows that the SPEA-II-based ATM performs significantly better as compared to the existing models.

show abstract

Section: Introductionmentioning

confidence: 99%

Medical Image Captioning Using Optimized Deep Learning Model

Singh

Raguru

Prasad

et al. 2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

show abstract

“…In the past few years, fully-supervised image captioning has been studied extensively [6]. The majority of the proposed models adopt the encoder-decoder paradigm where one Convolutional Neural Network (CNN) is leveraged to encode an input image firstly and one Recurrent Neural Network (RNN) is utilized to output a description for the image subsequently [1], [31], [33]. These models are trained to maximize the probability of generating the ground-truth captions, depending on enormous image-caption pairs.…”

Section: A Image Captioningmentioning

confidence: 99%

“…Similar to the standard image captioning models, the UIC model adopts the encoder-decoder framework [31] to encode an image into features and decode these features into captions. Any CNN backbone networks can be used for the feature extraction.…”

Section: E Unpaired Image Captioningmentioning

confidence: 99%

“…Finally, the predicted object category information and the relationship information are utilized to guide the unpaired image captioning. Given an 1 https://aws.amazon.com/sagemaker/groundtruth/pricing/ image, an encoder-decoder framework [31], [32] is adopted for caption generation, where an unrecognized object (UnO) loss is designed to integrate with the concept reward loss for optimization. The UnO loss contributes to excluding the unrecognized objects in the caption generation phase and therefore addresses the issue of unknown target generation, which has impacted negatively on existing UIC models.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Unpaired Image Captioning by Image-level Weakly-Supervised Visual Concept Recognition

Zhu¹,

Wang²,

Luo³

et al. 2022

Preprint

View full text Add to dashboard Cite

“…Image Captioning. Classical image captioning implements the encoder-decoder architecture, which first encodes images into features and decodes these image features into sentences [46,51,53] later. The goal of these models is to maximize the probability of generating the correct captions, relying on tremendous image-caption pairs [24].…”

Section: Related Workmentioning

confidence: 99%

Prompt-based Learning for Unpaired Image Captioning

Zhu¹,

Wang²,

Zhu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Unpaired Image Captioning (UIC) has been developed to learn image descriptions from unaligned vision-language sample pairs. Existing schemes usually adopt the visual concept reward of reinforcement learning to obtain the alignment between visual concepts and images. However, the cross-domain alignment is usually weak that severely constrains the overall performance of these existing schemes. Recent successes of Vision-Language Pre-Trained Models (VL-PTMs) have triggered the development of prompt-based learning from VL-PTMs. We present in this paper a novel scheme based on prompt to train the UIC model, making best use of the powerful generalization ability and abundant vision-language prior knowledge learned under VL-PTMs. We adopt the CLIP model for this research in unpaired image captioning. Specifically, the visual images are taken as input to the prompt generation module, which contains the pre-trained model as well as one feedforward layer for prompt extraction. Then, the input images and generated prompts are aggregated for unpaired adversarial captioning learning. To further enhance the potential performance of the captioning, we designed a high-quality pseudo caption filter guided by the CLIP logits to measure correlations between predicted captions and the corresponding images. This allows us to improve the captioning model in a supervised learning manner. Extensive experiments on the COCO and Flickr30K datasets have been carried out to validate the superiority of the proposed model. We have achieved the state-of-the-art performance on the COCO dataset, which outperforms the best UIC model by 1.9% on the BLEU-4 metric. We expect that the proposed prompt-based UIC model will inspire a new line of research for the VL-PTMs based captioning.

show abstract

Recall What You See Continually Using GridLSTM in Image Captioning

Cited by 35 publications

References 37 publications

Medical Image Captioning Using Optimized Deep Learning Model

Medical Image Captioning Using Optimized Deep Learning Model

Unpaired Image Captioning by Image-level Weakly-Supervised Visual Concept Recognition

Prompt-based Learning for Unpaired Image Captioning

Contact Info

Product

Resources

About