Unseen Object Instance Segmentation with Fully Test-time RGB-D Embeddings Adaptation

Abstract

Segmenting unseen objects is a crucial ability for the robot since it may encounter new environments during the operation. Recently, a popular solution is leveraging RGB-D features of large-scale synthetic data and directly applying the model to unseen real-world scenarios. However, even though depth data have fair generalization ability, the domain shift due to the Sim2Real gap is inevitable, which presents a key challenge to the unseen object instance segmentation (UOIS) model. To tackle this problem, we re-emphasize the adaptation process across Sim2Real domains in this paper. Specifically, we propose a framework to conduct the Fully Test-time RGB-D Embeddings Adaptation (FTEA) based on parameters of the BatchNorm layer. To construct the learning objective for test-time back-propagation, we propose a novel non-parametric entropy objective that can be implemented without explicit classification layers. Moreover, we design a cross-modality knowledge distillation module to encourage the information transfer during test time. The proposed method can be efficiently conducted with test-time images, without requiring annotations or revisiting the large-scale synthetic training data. Besides significant time savings, the proposed method consistently improves segmentation results on both overlap and boundary metrics, achieving state-of-the-art performances on two real-world RGB-D image datasets. We hope our work could draw attention to the test-time adaptation and reveal a promising direction for robot perception in unseen environments.

Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling

“…Despite achieving state-of-the-art UOIS performance, its application is limited by a long inference time of up to 10-15 s per frame. Zhang et al [31] proposed a test-time adaptation method to reduce the sim2real gap of UOIS methods; however, it requires approximately 20 seconds for new scenarios. In contrast, our approach facilitates fast refinement, processing frames in 0.1 s with comparable performance.…”

On Improving Boundary Quality of Instance Segmentation in Cluttered and Chaotic Scenarios

“…1 top-middle), i.e., testing SKUs different from training SKUs. While some recent works on UOIS [4], [5], [6], [7] start to address the seen-unseen domain gap, they may not work well for largescale storehouses with a huge amount and variety of SKUs.…”