An efficient algorithm for Co-segmentation

Abstract

This paper is focused on the Co-segmentation problem [1] -where the objective is to segment a similar object from a pair of images. The background in the two images may be arbitrary; therefore, simultaneous segmentation of both images must be performed with a requirement that the appearance of the two sets of foreground pixels in the respective images are consistent. Existing approaches [1,2] cast this problem as a Markov Random Field (MRF) based segmentation of the image pair with a regularized difference of the two histograms -assuming a Gaussian prior on the foreground appearance [1] or by calculating the sum of squared differences [2]. Both are interesting formulations but lead to difficult optimization problems, due to the presence of the second (histogram difference) term. The model proposed here bypasses measurement of the histogram differences in a direct fashion; we show that this enables obtaining efficient solutions to the underlying optimization model. Our new algorithm is similar to the existing methods in spirit, but differs substantially in that it can be solved to optimality in polynomial time using a maximum flow procedure on an appropriately constructed graph. We discuss our ideas and present promising experimental results.

Cosegmentation Revisited: Models and Optimization

“…Let us be more precise on the definition of the "same" foreground object. In this paper we use the definition of [1][2][3] where the only constraint is that the distribution of some appearance features of the foreground region in each image have to be similar. The appearance features can encode different information, like color and texture, and various similarity measures can be envisioned.…”

“…However, no comparison of such models and corresponding optimization techniques has been done so far. We analyze three existing models: the L1 norm model of Rother et al [1], the L2 norm model of Mukherjee et al [2] and the "reward" model of Hochbaum and Singh [3]. We also study a new model, which is a straightforward extension of the Boykov-Jolly model for single image segmentation [4].…”

Which Image Pairs Will Cosegment Well? Predicting Partners for Cosegmentation

“…At one end of the spectrum are methods that assume strong agreement in the inputs' foregrounds, i.e., that the two images contain the same exact object against differing backgrounds [1]. This setting continues to be developed, e.g., for greater efficiency [12] and multi-image collections with interactive user input [13]. In the middle of the spectrum is the weakly supervised scenario, where the input images are assumed to contain instances of the same object category [2,9,10,[3][4][5][6][7][8], and the goal is to extract the foreground per image (or possibly multiple foreground objects [6,7]).…”

“…In the strict same-object cosegmentation setting, this is assured by manually selecting the input pair (or set). For example, a designer may supply a set of images to be rotoscoped [1], or an analyst may gather aligned brain images from which to segment pathologies [12], or a consumer may group a burst of photos at an event (e.g., a soccer game) into a mini-album [13]. In the weakly supervised setting, the related images often originate from Internet search for an object's name.…”

“…1 Some work further relaxes the two-region (foreground/background) assumption to tackle k-region segmentation [11,7,6]. Furthermore, eager to capitalize on large collections of weakly labeled images, methods are being developed to account for both noisily labeled instances [11,8] and scalable optimization [12,7,6].…”