Bench-Marking And Improving Arabic Automatic Image Captioning Through The Use Of Multi-Task Learning Paradigm

Abstract

The continuous increase in the use of social media and the visual content on the internet have accelerated the research in computer vision field in general and the image captioning task in specific. The process of generating a caption that best describes an image is a useful task for various applications such as it can be used in image indexing and as a hearing aid for the visually impaired. In recent years, the image captioning task has witnessed remarkable advances regarding both datasets and architectures, and as a result, the captioning quality has reached an astounding performance. However, the majority of these advances especially in datasets are targeted for English, which left other languages such as Arabic lagging behind. Although Arabic language, being spoken by more than 450 million people and being the most growing language on the internet, lacks the fundamental pillars it needs to advance its image captioning research, such as benchmarks or unified datasets. This work is an attempt to expedite the synergy in this task by providing unified datasets and benchmarks, while also exploring methods and techniques that could enhance the performance of Arabic image captioning. The use of multi-task learning is explored, alongside exploring various word representations and different features. The results showed that the use of multi-task learning and pre-trained word embeddings noticeably enhanced the quality of image captioning, however the presented results show that Arabic captioning still lags behind when compared to the English language. The used dataset and code are available at this link.Keywords Image Captioning • Arabic Language • Multi-task Learning • Benchmark However, the overwhelming majority of the available datasets, algorithms, and research efforts were directed toward English, which leaves other languages lagging in this task. Arabic language for instance, is one of the majorly used languages on the web, used by more than 450 million people, and also among internet users, Arabic is the language with the fastest growth rate in the last few years [8]. Despite these facts, the image captions generation was untouched

A Systematic Literature Review on Using the Encoder-Decoder Models for Image Captioning in English and Arabic Languages

“…The image region vector is generated for each object detected and used as input to the final linear classification layer. The rest of the research depends on the CNN features, VggNet [13,15,18], ResNet [16,19,21], and Inception [18].…”

“…The experimental results showed promising performance. Al-muzaini et al [13] and ElJundi et al [15] use a single-layer LSTM with 256 memory units, while Za'ter et al [19] use two-layer LSTM with 256 memory units. Also, many other research works use LSTM, such as [14,16,17,21].…”

“…Afyouni et al [17] used GCT [101] to translate MSCOCO and Flicker8k datasets. Za'ter et al [19] translated using three open-source services: GCT [101], Facebook Machine Translation [102], and the University of Helsinki Open translation services [103]; the three datasets are MSCOCO, Flicker30k, and Flicker8k. Emami et al [20] and Lasheen et al [21] used two public databases, Arabic-COCO [104] and Arabic Flicker8k by ElJundi et al [15].…”

“…Thereafter, three articles were published in 2018 [12][13][14]. In 2020, two articles were published [15,16], in 2021, two articles were published [17,18] and three were published in 2022 [19][20][21]. Furthermore, some image captioning systematic literature reviews are available despite not covering some critical models between 2015 and 2023 [22,23].…”