CapNet: An Encoder-Decoder based Neural Network Model for Automatic Bangla Image Caption Generation

Abstract

Automatic caption generation from images has become an active research topic in the field of Computer Vision (CV) and Natural Language Processing (NLP). Machine generated image caption plays a vital role for the visually impaired people by converting the caption to speech to have a better understanding of their surrounding. Though significant amount of research has been conducted for automatic caption generation in other languages, far too little effort has been devoted to Bangla image caption generation. In this paper, we propose an encoder-decoder based model which takes an image as input and generates the corresponding Bangla caption as output. The encoder network consists of a pretrained image feature extractor called ResNet-50, while the decoder network consists of Bidirectional LSTMs for caption generation. The model has been trained and evaluated using a Bangla image captioning dataset named BanglaLekhaIm-ageCaptions. The proposed model achieved a training accuracy of 91% and BLEU-1, BLEU-2, BLEU-3, BLEU-4 scores of 0.81, 0.67, 0.57, and 0.51 respectively. Moreover, a comparative study for different pretrained feature extractors such as VGG-16 and Xception is presented. Finally, the proposed model has been deployed on an embedded device for analysing the inference time and power consumption.

A Comprehensive Survey on Image Captioning for Indian Languages: Techniques, Datasets, and Challenges

“…As opposed to traditional encoder-decoder architectures, Bahdanau Attention Based Bengali Image Caption Generation (BABBICG) [87] uses an InceptionV3 neural network to extract features from images and produce captions using an RNN decoder. [88] describe a model based on encoders and decoders that takes a picture as input and outputs the appropriate Bangla caption. The decoder network is made up of Bidirectional LSTMs for caption synthesis, whereas the encoder network is made up of ResNet-50, a pretrained image feature extractor.…”