Combining Image-Level and Segment-Level Models for Automatic Annotation

Abstract

Abstract. For the task of assigning labels to an image to summarize its contents, many early attempts use segment-level information and try to determine which parts of the images correspond to which labels. Best performing methods use global image similarity and nearest neighbor techniques to transfer labels from training images to test images. However, global methods cannot localize the labels in the images, unlike segment-level methods. Also, they cannot take advantage of training images that are only locally similar to a test image. We propose several ways to combine recent image-level and segment-level techniques to predict both image and segment labels jointly. We cast our experimental study in an unified framework for both image-level and segment-level annotation tasks. On three challenging datasets, our joint prediction of image and segment labels outperforms either prediction alone on both tasks. This confirms that the two levels offer complementary information.

Hybrid image representation methods for automatic image annotation: A survey

“…In addition, one image is represented by many visual feature vectors, resulting in high computational cost (see table 1). [45] More characteristic layout [53] Do not capture spatial information [54] Sensitive to intensity variations and distortion [37] Fail to narrow down the semantic gap due to their limited descriptive power based on objects [37] Do not have good performance [54] Weak in characterizing the internal content of image [49] Not recommended for multiple complex object images. Limited interpretability [53] …”

“…More precise, discriminating and explicit [53] [52] Spatial information [1] Good for search of specific object [52] [45] Improve classification [48] More flexible and Compositional character [53] Good generalization potential [53] High computational cost [49] High number of matches for a simple query [52] Need additional processing (e.g. segmentation) Not recommended when searching complex information [52] Produce unsatisfactory accuracy [49] …”