Learning similarity metrics for dynamic scene segmentation

Abstract

This paper addresses the segmentation of videos with arbitrary motion, including dynamic textures, using novel motion features and a supervised learning approach. Dynamic textures are commonplace in natural scenes, and exhibit complex patterns of appearance and motion (e.g. water, smoke, swaying foliage). These are difficult for existing segmentation algorithms, often violate the brightness constancy assumption needed for optical flow, and have complex segment characteristics beyond uniform appearance or motion. Our solution uses custom spatiotemporal filters that capture texture and motion cues, along with a novel metric-learning framework that optimizes this representation for specific objects and scenes. This is used within a hierarchical, graph-based segmentation setting, yielding state-of-the-art results for dynamic texture segmentation. We also demonstrate the applicability of our approach to general object and motion segmentation, showing significant improvements over unsupervised segmentation and results comparable to the best task specific approaches.

Dense Motion Estimation for Smoke

“…Right: Warping result using MDP method [39] (e.g. [30]). Finally, we designed the motion estimation method for smoke and have not yet tested it on other highly dynamics objects such as fire.…”

“…Algorithms exist that segment smoke and other natural phenomena from general backgrounds [30,31,32,33]. These methods provide various ways to detect areas with dynamic texture based on spatiotemporal filters [30], optical flow motion field [31]. But to focus on motion estimation alone, we use video acquired in laboratory conditions in which smoke appears to be light on a black background.…”

Fine‐grained recognition of maritime vessels and land vehicles by deep feature embedding

“…In numerous applications, variants of metric learning methods have been employed in solving several computer vision research problems including classification [28,33], person re-identification [38][39][40][41][42], zero-shot learning [43], and scene segmentation [44]. In [39], visual representations are adopted from a source domain to a target domain and then a similarity metric is learned to match images for person re-identification.…”

Multiscale Geoscene Segmentation for Extracting Urban Functional Zones from VHR Satellite Images

“…In other words, the definitions of pixels and objects do not match those of functional zones, leading to neither pixels nor objects representing functional zones (Table 1). For example, a pixel in Figure 3a is an imaging unit of a Existing image segmentation methods [9,17,18] are designed to extract homogeneous objects with consistent spectrums and regular shapes [19][20][21], but cannot delineate functional zones with high heterogeneities and substantial discontinuities in visual cues [22,23]. Considering the complexity and heterogeneity of functional zones, Heiden et al (2012) first proposed using road blocks to extract functional zones [24].…”

“…This is the first and fundamental step to functional-zone analysis. Existing segmentation methods can be sorted into three types: region, edge, and graph based [9,17,18]. Among them, a region-based method named multiresolution segmentation (MRS) outperforms others and is widely used for geographic-object-based analysis (GEOBIA) [18,37].…”