English to Hindi Multi-modal Neural Machine Translation and Hindi Image Captioning

Laskar, Sahinur Rahman; Singh, Rohit Pratap; Bandyopadhyay, Sivaji

doi:10.18653/v1/d19-5205

Cited by 16 publications

(8 citation statements)

References 15 publications

(14 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(Dutta Chowdhury et al, 2018) uses synthetic data, following multimodal NMT settings , and attains a BLEU score of 24.2 for Hindi to English translation. However, in the WAT 2019 multimodal translation task of English to Hindi, we achieved the highest BLEU score of 20.37 for the challenge test set (Laskar et al, 2019c). This score was improved later in the task of WAT2020 (Laskar et al, 2020c) and utilizes pre-train word embeddings of the monolingual corpus and additional parallel data of IITB.…”

Section: Related Workmentioning

confidence: 84%

“…For the English-Hindi language pair, the literature survey revealed minor existing works on translation using multimodal NMT (Dutta Chowdhury et al, 2018;Sanayai Meetei et al, 2019;Laskar et al, 2019c). (Dutta Chowdhury et al, 2018) uses synthetic data, following multimodal NMT settings , and attains a BLEU score of 24.2 for Hindi to English translation.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Improved English to Hindi Multimodal Neural Machine Translation

Laskar¹,

Khilji²,

Kaushik³

et al. 2021

Proceedings of the 8th Workshop on Asian Translation (WAT2021)

Self Cite

View full text Add to dashboard Cite

Machine translation performs automatic translation from one natural language to another. Neural machine translation attains a state-ofthe-art approach in machine translation, but it requires adequate training data, which is a severe problem for low-resource language pairs translation. The concept of multimodal is introduced in neural machine translation (NMT) by merging textual features with visual features to improve low-resource pair translation. WAT2021 (Workshop on Asian Translation 2021) organizes a shared task of multimodal translation for English to Hindi. We have participated the same with team name CNLP-NITS-PP in two submissions: multimodal and text-only translation. This work investigates phrase pairs injection via data augmentation approach and attains improvement over our previous work at WAT2020 on the same task in both text-only and multimodal translation. We have achieved second rank on the challenge test set for English to Hindi multimodal translation where Bilingual Evaluation Understudy (BLEU) score of 39.28, Rank-based Intuitive Bilingual Evaluation Score (RIBES) 0.792097, and Adequacy-Fluency Metrics (AMFM) score 0.830230 respectively.

show abstract

Section: Related Workmentioning

confidence: 84%

Section: Related Workmentioning

confidence: 99%

Improved English to Hindi Multimodal Neural Machine Translation

Laskar¹,

Khilji²,

Kaushik³

et al. 2021

Proceedings of the 8th Workshop on Asian Translation (WAT2021)

Self Cite

View full text Add to dashboard Cite

show abstract

“…A considerable amount of effort has been devoted to developing automatic image captioning techniques in languages such as English [1], Chinese [14], Japanese [15], Arabic [16], Hindi [17] and German [18] where large datasets related to image captioning are already available.…”

Section: Literature Reviewmentioning

confidence: 99%

CapNet: An Encoder-Decoder based Neural Network Model for Automatic Bangla Image Caption Generation

Rahman¹,

Murad²,

Rahman³

et al. 2022

IJACSA

View full text Add to dashboard Cite

Automatic caption generation from images has become an active research topic in the field of Computer Vision (CV) and Natural Language Processing (NLP). Machine generated image caption plays a vital role for the visually impaired people by converting the caption to speech to have a better understanding of their surrounding. Though significant amount of research has been conducted for automatic caption generation in other languages, far too little effort has been devoted to Bangla image caption generation. In this paper, we propose an encoder-decoder based model which takes an image as input and generates the corresponding Bangla caption as output. The encoder network consists of a pretrained image feature extractor called ResNet-50, while the decoder network consists of Bidirectional LSTMs for caption generation. The model has been trained and evaluated using a Bangla image captioning dataset named BanglaLekhaIm-ageCaptions. The proposed model achieved a training accuracy of 91% and BLEU-1, BLEU-2, BLEU-3, BLEU-4 scores of 0.81, 0.67, 0.57, and 0.51 respectively. Moreover, a comparative study for different pretrained feature extractors such as VGG-16 and Xception is presented. Finally, the proposed model has been deployed on an embedded device for analysing the inference time and power consumption.

show abstract

“…Moreover, P. Blandfort et al [32] systematically characterize diverse image captions that appear "in the wild" in order to understand how people caption images naturally. Alongside English researchers also generated captions in Chinese [15], [16], Japanese [1], Arabic [12], Bahasa Indonesia [13], Hindi [26] German [29] and Bengali [5], [23]. M. Rahman et al [23] generated image caption in Bengali for the first time followed by T. Deb et al [5].…”

Section: Related Workmentioning

confidence: 99%

A Hybridized Deep Learning Method for Bengali Image Captioning

Humaira¹,

Paul²,

Abidur³

et al. 2021

IJACSA

View full text Add to dashboard Cite

An omnipresent challenging research topic in computer vision is the generation of captions from an input image. Previously, numerous experiments have been conducted on image captioning in English but the generation of the caption from the image in Bengali is still sparse and in need of more refining. Only a few papers till now have worked on image captioning in Bengali. Hence, we proffer a standard strategy for Bengali image caption generation on two different sizes of the Flickr8k dataset and BanglaLekha dataset which is the only publicly available Bengali dataset for image captioning. Afterward, the Bengali captions of our model were compared with Bengali captions generated by other researchers using different architectures. Additionally, we employed a hybrid approach based on InceptionResnetV2 or Xception as Convolution Neural Network and Bidirectional Long Short-Term Memory or Bidirectional Gated Recurrent Unit on two Bengali datasets. Furthermore, a different combination of word embedding was also adapted. Lastly, the performance was evaluated using Bilingual Evaluation Understudy and proved that the proposed model indeed performed better for the Bengali dataset consisting of 4000 images and the BanglaLekha dataset.

show abstract

English to Hindi Multi-modal Neural Machine Translation and Hindi Image Captioning

Abstract: 20.37, Rank-based Intuitive Bilingual Evaluation Score (RIBES) 0.642838, Adequacy-Fluency Metrics (AMFM) score 0.668260 for challenge test data and BLEU score 40.55, RIBES 0.760080, AMFM score 0.770860 for evaluation test data in English to Hindi multimodal translation respectively.

Cited by 16 publications

References 15 publications

Improved English to Hindi Multimodal Neural Machine Translation

Improved English to Hindi Multimodal Neural Machine Translation

CapNet: An Encoder-Decoder based Neural Network Model for Automatic Bangla Image Caption Generation

A Hybridized Deep Learning Method for Bengali Image Captioning

Contact Info

Product

Resources

About