Neuromorphic Event-Based Generalized Time-Based Stereovision

Abstract

3D reconstruction from multiple viewpoints is an important problem in machine vision that allows recovering tridimensional structures from multiple two-dimensional views of a given scene. Reconstructions from multiple views are conventionally achieved through a process of pixel luminance-based matching between different views. Unlike conventional machine vision methods that solve matching ambiguities by operating only on spatial constraints and luminance, this paper introduces a fully time-based solution to stereovision using the high temporal resolution of neuromorphic asynchronous event-based cameras. These cameras output dynamic visual information in the form of what is known as “change events” that encode the time, the location and the sign of the luminance changes. A more advanced event-based camera, the Asynchronous Time-based Image Sensor (ATIS), in addition of change events, encodes absolute luminance as time differences. The stereovision problem can then be formulated solely in the time domain as a problem of events coincidences detection problem. This work is improving existing event-based stereovision techniques by adding luminance information that increases the matching reliability. It also introduces a formulation that does not require to build local frames (though it is still possible) from the luminances which can be costly to implement. Finally, this work also introduces a methodology for time based stereovision in the context of binocular and trinocular configurations using time based event matching criterion combining for the first time all together: space, time, luminance, and motion.

Event-Based Visual Inertial Odometry

“…To prove the effectiveness of the proposed mapping method, we compare against three stereo methods and ground truth depth when available. The baseline methods are abbreviated by GTS [26], SGM [45] and CopNet [63]. a) Description of Baseline Methods: The method in [26] proposes to match events by using a per-event time-based consistency criterion that also works on grayscale events from the [27] camera; after that, classical triangulation provides the 3D point location.…”

“…The baseline methods are abbreviated by GTS [26], SGM [45] and CopNet [63]. a) Description of Baseline Methods: The method in [26] proposes to match events by using a per-event time-based consistency criterion that also works on grayscale events from the [27] camera; after that, classical triangulation provides the 3D point location. Since the code for this method is not available, we implement an abridged version of it, without the term for grayscale events because they are not available with the DAVIS.…”

“…Event matching is often solved by enforcing several constraints, including temporal coherence (e.g., simultaneity) of events across both cameras. For example, [26] combines epipolar constraints, temporal inconsistency, motion inconsistency and photometric error (available only from grayscale events given by ATIS cameras [27]) into an objective function to compute the best matches. Other works, such as [28], [29], [30], extend cooperative stereo [31] to the case of event cameras [32].…”

“…Remark on Modeling Data Association: Note that our approach differs from classical two-step event-processing methods [22], [23], [24], [26] that solve the stereo matching problem first and then triangulate the 3D point. Such two-step approaches work in a "back-projection" fashion, mapping 2D event measurements into 3D space.…”

“…Inverse depth by GTS [26] Inverse depth by SGM [45] Inverse depth by CopNet [63] Inverse depth (Ours) The first column shows intensity frames from the DAVIS camera (not used, just for visualization). Columns 2 to 5 show inverse depth estimation results of GTS [26], SGM [45], CopNet [63] and our method, respectively.…”