Image Annotation by a Hierarchical and Iterative Combination of Recognition and Segmentation

Abstract

Automatic image annotation and image segmentation are two prominent research fields of Computer Vision, that are getting higher attention these days to accomplish image analysis and scene understanding. In this work, we present an annotation algorithm based on a hierarchical image partition, that makes use of Markov Random Fields (MRFs) to model spatial and hierarchical relations among regions in the image. In this way, we can capture local, global and contextual information. Also, we combine the processes of annotation and segmentation in an iterative way so that each process benefits from the other. Furthermore, we investigate the selection of the starting segmentation level for the hierarchical annotation process, to show its relevance for the final results. We experimentally validate our approach in three well-known datasets: CorelA, Stanford Background and MSRC-21 datasets. In these datasets, we achieved better or comparable results to other state-of-the-art algorithms, improving our base classifier in all cases.

Multi-hypothesis contextual modeling for semantic segmentation

“…Vieux et al (2012) also labels segments by late fusion of SVM classifiers over multiple segmentations; however, fusion is simply performed by taking the mean/max/multiplication of classifier probabilities in intersecting regions and label smoothing by relaxation labeling is treated as a post-processing step on the fused result. Methods such as Dong et al (2016), Yao et al (2012), Morales-Gonzalez et al (2018) define hierarchical MRF models over multiple segmentations but do not consider segmentations and class scores coming from alternative methods. In these approaches, since the segmentations and their unary potentials at different levels of the hierarchy are not independently generated, there will be no significant complementary information for fusion over the hierarchical MRF.…”