Quality Estimation for Image Captions Based on Large-scale Human Evaluations

Abstract

Automatic image captioning has improved significantly over the last few years, but the problem is far from being solved, with state of the art models still often producing low quality captions when used in the wild. In this paper, we focus on the task of Quality Estimation (QE) for image captions, which attempts to model the caption quality from a human perspective and without access to groundtruth references, so that it can be applied at prediction time to detect low-quality captions produced on previously unseen images. For this task, we develop a human evaluation process that collects coarse-grained caption annotations from crowdsourced users, which is then used to collect a large scale dataset spanning more than 600k caption quality ratings. We then carefully validate the quality of the collected ratings and establish baseline models for this new QE task. Finally, we further collect fine-grained caption quality annotations from trained raters, and use them to demonstrate that QE models trained over the coarse ratings can effectively detect and filter out lowquality image captions, thereby improving the user experience from captioning systems.

IC3: Image Captioning by Committee Consensus

“…Significant prior work has explored the collection of human judgments of the quality of visual descriptions. Human judgment is considered the gold standard for visual description evaluation, and previous studies typically rely on human annotators to rate caption quality on one or multiple axes (Levinboim et al, 2021;Kasai et al, 2022). While automated methods exist for the evaluation of caption quality (Agarwal and Lavie, 2008;Vedantam et al, 2015;Papineni et al, 2002), recent work including THUMB (Kasai et al, 2022), which has run human evaluations on captions produced by models based on "Precision", "Recall", "Fluency", "Conciseness" and "Inclusive Language", has shown that humans produce captions which score significantly higher when judged by human raters, than when judged by existing measures (and further, that human judgments of quality correlate poorly with existing measures), necessitating the need for human evaluation as opposed to evaluation of captioning methods using automated measures for caption quality.…”

#PraCegoVer: A Large Dataset for Image Captioning in Portuguese

“…Many large-scale datasets have been created [6,[10][11][12][13][14][15][16][17][18][19][20], but in contrast to the previous ones, they employ automated pipelines. One example of a dataset that follows this approach is the Conceptual Captions dataset [5] which has more than 3.3M pairs of images and English captions.…”

“…[1][2][3][4][5][6][7][8][9][10],, [101-1000], [1001-10,000],[10,[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20],000], and 20,001 or more.…”

UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learning

“…QE is widely established in machine translation (MT) tasks (Specia et al, 2013;Martins et al, 2017;Specia et al, 2018). Recently, (Levinboim et al, 2021) introduces a large scale human ratings on image-caption pairs for training QE models in image captioning tasks. Our work also trains caption QE model, (i.e.…”