VLSP 2021 - VieCap4H Challenge: Automatic Image Caption Generation for Healthcare Domain in Vietnamese

Abstract

This paper presents VieCap4H, a grand data challenge on automatic image caption generation for the healthcare domain in Vietnamese. VieCap4H is held as part of the eighth annual workshop on VietnameseLanguage and Speech Processing (VLSP 2021). The task is considered as an image captioning task. Given a static image, mostly about healthcare-related scenarios, participants are asked to design machine learning methods to generate natural language captions in Vietnamese to describe the visual content of the image. We introduce VieCap4H, a novel human-annotated image captioning dataset in Vietnamese that contains over 10,000 image-caption pairs collected from real-world scenarios in the healthcare domain. All the models proposed by the challenge participants are evaluated using BLEU scores against groundtruths. The challenge was run on AIHUB.VN platform. Within less than two months, the challenge has attracted over 90 individual participants and recorded more than 900 valid submissions.

vieCap4H Challenge 2021: A transformer-based method for Healthcare Image Captioning in Vietnamese

“…In Transformer architecture, we use 𝑁 𝑒𝑛𝑐𝑜𝑑𝑒𝑟 = 3, 𝑁 𝑑𝑒𝑐𝑜𝑑𝑒𝑟 = 3, 𝑑 𝑚𝑜𝑑𝑒𝑙 = 512 and the number of attention heads is 8. During inference, we use three values 𝑚𝑎𝑥𝑙𝑒𝑛𝑔𝑡ℎ ∈ [20,22,23], apply beam search with 𝑏𝑒𝑎𝑚𝑠𝑖𝑧𝑒 ∈ [3,4,5].and use 50 submissions in public-test round to evaluate. For the private test, we use two values 𝑚𝑎𝑥𝑙𝑒𝑛𝑔𝑡ℎ ∈ [22,23,24].…”

“…For evaluation, we use the Bilingual Evaluation Understudy (BLEU) metric, which was first used for evaluating the performance of the captioning model in [1]. This metric is also used in the VieCap4H challenge [3]. BLEU score is commonly used in machine translation tasks.…”

“…Inside, the feature extraction problem in image captioning has two main approaches: grid features and region features; which one is more effective are now still being discussed. On the other hand, the VieCap4H [3] dataset is a Vietnamese dataset, so mining the aspect of the Vietnamese language by using a BERT-like model with Vietnamese pre-trained models such as PhoBERT, BARTPho may achieve good results.…”

vieCap4H Challenge 2021: Vietnamese Image Captioning for Healthcare Domain using Swin Transformer and Attention-based LSTM

“…To encourage conducting research on Vietnamese image captioning, [2] created a dataset for Vietnamese domain, also serving as a premise for researching on Vietnamese image captioning for healthcare domain. The vieCap4H Challenge 2021 [3] aims to be a competition for developing machine learning algorithms that use Vietnamese to describe the visual content in healthcare settings, especially images that describe the COVID-19 pandemic. Similar to this task, the most recent studies were presented in [4] and [5] that proposed a network involving a deep Convolutional Neural Network (CNN) as an encoder and a Recurrent Neural Network (RNN) as a decoder.…”

Empirical Study of Feature Extraction Approaches for Image Captioning in Vietnamese

“…To evaluate the performance of different types of region features, Transformer-based [35] model is used to train to generating captions. Two benchmark datasets for image captioning in Vietnamese are used to evaluate the effectiveness: UIT-ViIC [13] and VieCap4H [14].…”