Learning to Refine Object Segments

Abstract

Abstract. Object segmentation requires both object-level information and low-level pixel data. This presents a challenge for feedforward networks: lower layers in convolutional nets capture rich spatial information, while upper layers encode object-level knowledge but are invariant to factors such as pose and appearance. In this work we propose to augment feedforward nets for object segmentation with a novel top-down refinement approach. The resulting bottom-up/top-down architecture is capable of efficiently generating high-fidelity object masks. Similarly to skip connections, our approach leverages features at all layers of the net. Unlike skip connections, our approach does not attempt to output independent predictions at each layer. Instead, we first output a coarse 'mask encoding' in a feedforward pass, then refine this mask encoding in a topdown pass utilizing features at successively lower layers. The approach is simple, fast, and effective. Building on the recent DeepMask network for generating object proposals, we show accuracy improvements of 10-20% in average recall for various setups. Additionally, by optimizing the overall network architecture, our approach, which we call SharpMask, is 50% faster than the original DeepMask network (under .8s per image).

DeepProposals: Hunting Objects and Actions by Cascading Deep Convolutional Layers

“…it dedicates a network just for generating proposals. DeepMask (Pinheiro et al 2015) and its next generation, SharpMask (Pinheiro et al 2016), learn segmentation proposals by training a network to predict a class-agnostic mask for each image patch and an associated score. Similar to us, SharpMask uses a top-down refinement approach, utilizing features at lower layers to refine and generate segmentation masks with double the spatial resolution.…”

Learning to Co-Generate Object Proposals with a Deep Structured Network

“…Recently, Pinheiro et al [18] proposed to go beyond handcrafted features for object proposal generation. In particular, [18] leverages the representation power of deep networks to learn a discriminative CNN that generates boxes and segmentation proposals.…”

“…In particular, [18] leverages the representation power of deep networks to learn a discriminative CNN that generates boxes and segmentation proposals. Ultimately, this method achieves state-of-the-art accuracy and competitive speed.…”

“…For Pascal VOC 2007, we used all the trainval data to learn our deep structured model, and evaluated it using all the test data. For Microsoft COCO, we also used all the training images to learn our model, but, following [18], used only the first 5000 validation images for evaluation purpose.…”

“…Existing object proposal methods [1,4,15,18,22,25,12], however, all generate candidate detections one by one, independently of each other. By contrast, in a parallel line of research, object co-detection [2,8,10,11] has emerged as an effective approach to leveraging the information jointly contained in multiple images to improve detection accuracy.…”