SPICE: Semantic Propositional Image Caption Evaluation

Abstract

There is considerable interest in the task of automatically generating image captions. However, evaluation is challenging. Existing automatic evaluation metrics are primarily sensitive to n-gram overlap, which is neither necessary nor sufficient for the task of simulating human judgment. We hypothesize that semantic propositional content is an important component of human caption evaluation, and propose a new automated caption evaluation metric defined over scene graphs coined SPICE. Extensive evaluations across a range of models and datasets indicate that SPICE captures human judgments over model-generated captions better than other automatic metrics (e.g., system-level correlation of 0.88 with human judgments on the MS COCO dataset, versus 0.43 for CIDEr and 0.53 for METEOR). Furthermore, SPICE can answer questions such as which caption-generator best understands colors? and can caption-generators count?

A hierarchical and regional deep learning architecture for image description generation

“…BLEU-1, BLEU-2, BLEU-3 and BLEU-4) based on the n-gram method of determining string/sentence similarity. It is also equipped with other evaluation metrics such as, METEOR [31], ROUGE-L [32], and SPICE [33]. The detailed results using all of the above metrics for the evaluation of the IAPR TC-12 dataset are shown in Table 3.…”

REO-Relevance, Extraness, Omission: A Fine-grained Evaluation for Image Captioning

“…Following standard practice (Anderson et al, 2016;Elliott and Keller, 2014), we compared with Text in red is extra information, while text in green is missing information. sim(x, y) is the average similarity between machine-identified and true error vectors over image regions.…”

Movie Description

“…19 The automatic evaluation measures include BLEU-1,-2,-3,-4 (Papineni et al 2002), METEOR (Denkowski and Lavie 2014), ROUGE-L (Lin 2004), and CIDEr . We also use the recently proposed evaluation measure SPICE (Anderson et al 2016), which aims to compare the semantic content of two descriptions, by matching the information contained in dependency parse trees for both descriptions. While we report all measures for the final evaluation in the LSMDC (Sect.…”