A Survey on Bengali Image Captioning: Architectures, Challenges, and Directions

Sukhi, Shamima; Ohi, Abu Quwsar; Rahman, Md. Saifur; Mridha, M. F.

doi:10.1109/icsct53883.2021.9642554

Cited by 4 publications

(2 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…All three works have used a slightly modified version of the encoder and decoder architecture proposed by (Vinyals et al, 2015 [1]). The application of image captioning to Indian languages was initially explored in pioneering work on the Hindi language (Dhir et al, 2019 [6]), followed by subsequent research on the Bengali language (Rahman et al, 2019 [25]). From that beginning numerous advancements have been made in the Hindi and Bengali languages.…”

Section: Image Captioning In Other Languagesmentioning

confidence: 99%

Multilingual Image Captioning: Multimodal Framework for Bridging Visual and Linguistic Realms in Tamil and Telugu through Transformers

Gunti,

et al. 2023

Preprint

View full text Add to dashboard Cite

Image captioning is a formidable challenge in the realms of computer vision and natural language processing. The limited availability of captioning mechanisms for non-English languages poses significant barriers for nonproficient English speakers. Specifically, Tamil and Telugu the most spoken languages in India, lack image captioning models capable of delivering accurate captions. Moreover, generating captions in Tamil and Telugu is a complex task due to their unique linguistic intricacies. Addressing these challenges requires advanced models capable of capturing long-range dependencies and generating contextually meaningful image captions. This research presents a multimodal deep learning framework that appropriately integrates InceptionV3, VGG16, and ResNet50 convolutional neural network architectures with multihead attention-based transformer architecture. By harnessing the multihead attention mechanism, our model effectively comprehends image context, handles linguistic complexity, and establishes vital multimodal associations between visual and textual features. Extensive experiments were carried out on translated versions of the benchmark datasets such as Flickr8k, Flickr30k, and MSCOCO to evaluate the efficacy of the model. The multimodal technique we suggested produced extraordinary and remarkable results especially in terms of BLEU metrics. The maximum value of the BLEU-1 score achieved by our suggested model reaches 65.16 and 66.79 on Tamil and Telugu caption generation tasks respectively. These findings outperformed the results of existing methods, indicating improved accuracy in generating captions for both Tamil and Telugu languages. Furthermore, a meticulous, manually labored audit of the generated captions confirmed their appropriateness and competence, affirming the robustness of the proposed methodology.

show abstract

Section: Image Captioning In Other Languagesmentioning

confidence: 99%

Multilingual Image Captioning: Multimodal Framework for Bridging Visual and Linguistic Realms in Tamil and Telugu through Transformers

Gunti,

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…This task unites computer vision and natural language processing methodologies to produce an easily understandable narrative that concisely conveys the image's content. The primary objective is to offer a brief and precise depiction of the elements, settings, actions, and occurrences depicted in the image [1][2][3]. There is an increasing daily demand for image retrieval and analysis systems because they are used in many fields and on a large scale via the Internet, social media, and various search engines [4] [5].…”

Section: Introductionmentioning

confidence: 99%

Automatic Generation of Image Caption Based on Semantic Relation using Deep Visual Attention Prediction

EL-GAYAR

2023

IJACSA

View full text Add to dashboard Cite

While modern systems for managing, retrieving, and analyzing images heavily rely on deriving semantic captions to categorize images, this task presents a considerable challenge due to the extensive capabilities required for manual processing, particularly with large images. Despite significant advancements in automatic image caption generation and human attention prediction through convolutional neural networks, there remains a need to enhance attention models in these networks through efficient multi-scale features utilization. Addressing this need, our study presents a novel image decoding model that integrates a wavelet-driven convolutional neural network with a dual-stage discrete wavelet transform, enabling the extraction of salient features within images. We utilize a wavelet-driven convolutional neural network as the encoder, coupled with a deep visual prediction model and Long Short-Term Memory as the decoder. The deep Visual Prediction Model calculates channel and location attention for visual attention features, with local features assessed by considering the spatial-contextual relationship among objects. Our primary contribution is to propose an encoder and decoder model to automatically create a semantic caption on the image based on the semantic contextual information and spatial features present in the image. Also, we improved the performance of this model, demonstrated through experiments conducted on three widely used datasets: Flickr8K, Flickr30K, and MSCOCO. The proposed approach outperformed current methods, achieving superior results in BLEU, METEOR, and GLEU scores. This research offers a significant advancement in image captioning and attention prediction models, presenting a promising direction for future work in this field.

show abstract