0-MMS: Zero-Shot Multi-Motion Segmentation With A Monocular Event Camera

Abstract

Segmentation of moving objects in dynamic scenes is a key process in scene understanding for both navigation and video recognition tasks. Without prior knowledge of the object structure and motion, the problem is very challenging due to the plethora of motion parameters to be estimated while being agnostic to motion blur and occlusions. Event sensors, because of their high temporal resolution, and lack of motion blur, seem well suited for addressing this problem. We propose a solution to multi-object motion segmentation using a combination of classical optimization methods along with deep learning and does not require prior knowledge of the 3D motion and the number and structure of objects. Using the events within a time-interval, the method estimates and compensates for the global rigid motion. Then it segments the scene into multiple motions by iteratively fitting and merging models using as input tracked feature regions via alignment based on temporal gradients and contrast measures. The approach was successfully evaluated on both challenging real-world and synthetic scenarios from the EV-IMO, EED and MOD datasets and outperforms the state-of-the-art detection rate by as much as 12% achieving a new state-of-the-art average detection rate of 77.06%, 94.2% and 82.35% on the aforementioned datasets.

Event-based Motion Segmentation by Cascaded Two-Level Multi-Model Fitting

“…1(c)). The binary classification task is formulated as a 2 We observed no apparent difference in performance whether or not a global motion compensation is performed as pre-processing [13,14].…”

“…More recently, a hierarchical clustering method was presented in [13] and further improved in [14]. The method first clustered event features by applying the K-means method.…”

“…Unlike [15], we propose a cascaded two-level multi-model fitting scheme. In the first level, we bring in event feature tracking [13,14,17] for efficient hypothesis proposal. Instead of using K-means [13,14], we cluster the features with a progressive multi-model fitting method.…”

“…In the first level, we bring in event feature tracking [13,14,17] for efficient hypothesis proposal. Instead of using K-means [13,14], we cluster the features with a progressive multi-model fitting method. The model fitting and featuremotion association sub-problems are solved jointly, thus not requiring incremental cluster merging locally [13,14].…”

“…Instead of using K-means [13,14], we cluster the features with a progressive multi-model fitting method. The model fitting and featuremotion association sub-problems are solved jointly, thus not requiring incremental cluster merging locally [13,14]. In the second level, we leverage the spatio-temporal graph cut method [15] to solve the event clustering problem.…”