Semantic Video Segmentation from Occlusion Relations within a Convex Optimization Framework

Abstract

Abstract. We describe an approach to incorporate scene topology and semantics into pixel-level object detection and localization. Our method requires video to determine occlusion regions and thence local depth ordering, and any visual recognition scheme that provides a score at local image regions, for instance object detection probabilities. We set up a cost functional that incorporates occlusion cues induced by object boundaries, label consistency and recognition priors, and solve it using a convex optimization scheme. We show that our method improves localization accuracy of existing recognition approaches, or equivalently provides semantic labels to pixel-level localization and segmentation.

Optical Flow with Semantic Segmentation and Localized Layers

“…They usually do not use high-level object recognizers or try to improve optical flow. Taylor et al [51] incorporate object detections and use temporal information to reason about occlusions to improve their segmentation results, but do not compute optical flow. Lalos et al [30] compute optical flow for an object of interest using a tracking-by-detection approach.…”

Multi-class Semantic Video Segmentation with Exemplar-Based Object Reasoning

“…However, they do not model multiple semantic object classes, nor do they capture contextual relations between objects and background classes. Taylor et al [21] jointly infer pixel semantic classes and occlusion relationship in video segmentation. Unlike our method, they do not incorporate object instance level reasoning.…”

Visual Information Theory and Visual Representation for Achieving Provable Bounds in Vision-Based Control and Decision

“…It also provides relations between objects, in the sense that the scheme does not just attach labels to object, but also determines whether there are multiple objects, and in what depth ordering they are presented relative to the viewer. The key results have been presented in [17], where the ideas have been tested on benchmark datasets.…”

“…The following references describe work that has been conducted during this project and acknowledge support by ARO: Year 1: [14,3,9,15,1,5,8,12], Year 2: [2,4,10,6,11,19], Year 3: [7,17,18].…”

“…While complete transitions have not been accomplished during the period of performance, since the research focused on fundamental issues underlying the theoretical development of a theory of visual information, the research milestones accomplished enabled improvement of specific tasks that we envision will result in transitions in the near to mid-term future. These include visual-inertial sensor fusion [18], and semantic video segmentation [17].…”