A Hybridized Deep Learning Method for Bengali Image Captioning

Abstract

An omnipresent challenging research topic in computer vision is the generation of captions from an input image. Previously, numerous experiments have been conducted on image captioning in English but the generation of the caption from the image in Bengali is still sparse and in need of more refining. Only a few papers till now have worked on image captioning in Bengali. Hence, we proffer a standard strategy for Bengali image caption generation on two different sizes of the Flickr8k dataset and BanglaLekha dataset which is the only publicly available Bengali dataset for image captioning. Afterward, the Bengali captions of our model were compared with Bengali captions generated by other researchers using different architectures. Additionally, we employed a hybrid approach based on InceptionResnetV2 or Xception as Convolution Neural Network and Bidirectional Long Short-Term Memory or Bidirectional Gated Recurrent Unit on two Bengali datasets. Furthermore, a different combination of word embedding was also adapted. Lastly, the performance was evaluated using Bilingual Evaluation Understudy and proved that the proposed model indeed performed better for the Bengali dataset consisting of 4000 images and the BanglaLekha dataset.

CapNet: An Encoder-Decoder based Neural Network Model for Automatic Bangla Image Caption Generation

“…In literature, vision encoder is designed using stacked Convolutional Neutral Network (CNN) [1], and graph-based network [2]. Moreover, various pre-trained feature extractors such as VGG-16, InceptionResnetV2, and Xception have been deployed for vision encoding [3], [12] The language decoder is implemented using variations of Recurrent Neural Networks (RNNs) such as LSTMs and GRUs [2]. In addition, self attention based transformer models are utilised to design the language decoder [13].…”

“…Researchers have not addressed automatic image captioning in Bangla for a long period of time due to a lack of an enriched dataset. After development of required dataset, several researches have been conducted on Bangla caption generation from visual image [3], [4], [5], [6], [7].…”

Bornon: Bengali Image Captioning with Transformer-Based Deep Learning Approach

“…Furthermore, [12] utilized the BNLIT dataset to implement a CNN-RNN model where they used both BRNN and LSTM as RNN. Humaira et al [3] proposed a hybridized encoder-decoder approach where two word embeddings fastText and GloVe were concatenated. They also utilized beam search and greedy search to compute the BLEU scores.…”

“…For image captioning in Bengali, those 40455 captions were converted to Bengali language using Google Translator. 3 Unfortunately, some of the translated captions were syntactically incorrect as shown in Fig. 5.…”

“…In this task, the encoder is used to extract the image feature to obtain feature vectors, then pass it through an RNN to generate the language description. Previously, all researchers utilized this CNN-RNN [1][2][3] approach to generate captions from images. However, this method has a drawback, that is, due to the structure of the LSTM or other RNNs, the current output depends on the hidden state at the previous moment.…”