Unknown Object Segmentation from Stereo Images

Abstract

Although instance-aware perception is a key prerequisite for many autonomous robotic applications, most of the methods only partially solve the problem by focusing solely on known object categories. However, for robots interacting in dynamic and cluttered environments, this is not realistic and severely limits the range of potential applications. Therefore, we propose a novel object instance segmentation approach that does not require any semantic or geometric information of the objects beforehand. In contrast to existing works, we do not explicitly use depth data as input, but rely on the insight that slight viewpoint changes, which for example are provided by stereo image pairs, are often sufficient to determine object boundaries and thus to segment objects. Focusing on the versatility of stereo sensors, we employ a transformer-based architecture that maps directly from the pair of input images to the object instances. This has the major advantage that instead of a noisy, and potentially incomplete depth map as an input, on which the segmentation is computed, we use the original image pair to infer the object instances and a dense depth map. In experiments in several different application domains, we show that our Instance Stereo Transformer (INSTR) algorithm outperforms current state-of-the-art methods that are based on depth maps. Training code and pretrained models are available at https://github.com/DLR-RM/instr.

Challenges of SLAM in Extremely Unstructured Environments: The DLR Planetary Stereo, Solid-State LiDAR, Inertial Dataset

“…In order to illustrate what may lead to such methods, we employed the Instance Stereo Transformer (INSTR) [47], to detect instances of unknown objects in a scene (see Fig. 6 for two examples).…”

“…Fig.6. Examples of instance segmentation of unknown objects, in this case small stones, using INSTR[47]. Shown are original left camera images (left column) and instance predictions as colored overlays (right column).…”

Bayesian Active Learning for Sim-to-Real Robotic Perception

“…Jianxiang Feng 1,2 , Jongseok Lee 1 , Maximilian Durner 1,2 and Rudolph Triebel 1,2 Abstract-While learning from synthetic training data has recently gained an increased attention, in real-world robotic applications, there are still performance deficiencies due to the so-called Sim-to-Real gap. In practice, this gap is hard to resolve with only synthetic data.…”

“…Over the last years, the performance of computer vision increased sharply, leading to the urge of employing such approaches on robotic vision tasks such as object classification, detection [1], [2] and pose estimation [3]. In this context, the necessity of large amounts of annotated, task-related training data is a main issue, particularly for tasks relying on semantic features such as object classification or detection.…”

Template-Based Category-Agnostic Instance Detection for Robotic Manipulation

“…[38] performed category-agnostic segmentation on the synthesized dataset using the mask R-CNN with domain randomization. Similarly, stereo images were used in [39], which can also be used to segment unknown object instances on generic horizontal surfaces. However, these methods cannot select a specific instance from segmented results.…”