Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling

Back, Seunghyeok; Lee, Joosoon; Kim, Tae−Won; Noh, Sangjun; Kang, Raeyoung; Bak, Seongho; Lee, Kyoobin

doi:10.48550/arxiv.2109.11103

Cited by 4 publications

(8 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, RICE [3] focuses on the occlusion problem in clutter scenes and utilizes a graphbased representation of instance masks to refine the outputs of previous methods. UOAIS-Net [15] presents a new unseen object amodal instance segmentation (UOAIS) task to emphasize the amodal perception for robotic manipulation in a cluttered scene. Besides, UOAIS-Net introduce a largescale photo-realistic synthetic dataset named UOAIS-SIM to improve the sim2real transferability.…”

Section: Related Workmentioning

confidence: 99%

Unseen Object Instance Segmentation with Fully Test-time RGB-D Embeddings Adaptation

Zhang¹,

Zhang²,

Xu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Segmenting unseen objects is a crucial ability for the robot since it may encounter new environments during the operation. Recently, a popular solution is leveraging RGB-D features of large-scale synthetic data and directly applying the model to unseen real-world scenarios. However, even though depth data have fair generalization ability, the domain shift due to the Sim2Real gap is inevitable, which presents a key challenge to the unseen object instance segmentation (UOIS) model. To tackle this problem, we re-emphasize the adaptation process across Sim2Real domains in this paper. Specifically, we propose a framework to conduct the Fully Test-time RGB-D Embeddings Adaptation (FTEA) based on parameters of the BatchNorm layer. To construct the learning objective for test-time back-propagation, we propose a novel non-parametric entropy objective that can be implemented without explicit classification layers. Moreover, we design a cross-modality knowledge distillation module to encourage the information transfer during test time. The proposed method can be efficiently conducted with test-time images, without requiring annotations or revisiting the large-scale synthetic training data. Besides significant time savings, the proposed method consistently improves segmentation results on both overlap and boundary metrics, achieving state-of-the-art performances on two real-world RGB-D image datasets. We hope our work could draw attention to the test-time adaptation and reveal a promising direction for robot perception in unseen environments.

show abstract

Section: Related Workmentioning

confidence: 99%

Unseen Object Instance Segmentation with Fully Test-time RGB-D Embeddings Adaptation

Zhang¹,

Zhang²,

Xu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This problem, with accompanying datasets, was introduced by Zhu et al [21]. UOIS 2D [22] and 3D [23] and UOAIS-Net [24] utilize RGB-D images to do so. As with the current works on stacking hierarchy inference, these works do not reason about the uncertainty concerning the stacking hierarchy.…”

Section: Related Workmentioning

confidence: 99%

DUQIM-Net: Probabilistic Object Hierarchy Representation for Multi-View Manipulation

Turetsky¹,

Miron²,

Di-Castro³

2022

Preprint

View full text Add to dashboard Cite

Object manipulation in cluttered scenes is a difficult and important problem in robotics. To efficiently manipulate objects, it is crucial to understand their surroundings, especially in cases where multiple objects are stacked one on top of the other, preventing effective grasping. We here present DUQIM-Net, a decision-making approach for object manipulation in a setting of stacked objects. In DUQIM-Net, the hierarchical stacking relationship is assessed using Adj-Net, a model that leverages existing Transformer Encoder-Decoder object detectors by adding an adjacency head. The output of this head probabilistically infers the underlying hierarchical structure of the objects in the scene. We utilize the properties of the adjacency matrix in DUQIM-Net to perform decision making and assist with object-grasping tasks. Our experimental results show that Adj-Net surpasses the state-of-the-art in object-relationship inference on the Visual Manipulation Relationship Dataset (VMRD), and that DUQIM-Net outperforms comparable approaches in bin clearing tasks.

show abstract

“…1 top-middle), i.e., testing SKUs different from training SKUs. While some recent works on UOIS [4], [5], [6], [7] start to address the seen-unseen domain gap, they may not work well for largescale storehouses with a huge amount and variety of SKUs.…”

Section: Introductionmentioning

confidence: 99%

On Improving Boundary Quality of Instance Segmentation in Cluttered and Chaotic Scenarios

Yang

Gao

et al. 2023

2023 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

In large-scale storehouses, precise instance masks are crucial for robotic bin picking but are challenging to obtain. Existing instance segmentation methods typically rely on a tedious process of scene collection, mask annotation, and network fine-tuning for every single Stock Keeping Unit (SKU). This paper presents SKU-Patch, a new patch-guided instance segmentation solution, leveraging only a few image patches for each incoming new SKU to predict accurate and robust masks, without tedious manual effort and model re-training. Technical-wise, we design a novel transformer-based network with (i) a patch-image correlation encoder to capture multi-level image features calibrated by patch information and (ii) a patchaware transformer decoder with parallel task heads to generate instance masks. Extensive experiments on four storehouse benchmarks manifest that SKU-Patch is able to achieve the best performance over the state-of-the-art methods. Also, SKU-Patch yields an average of nearly 100% grasping success rate on more than 50 unseen SKUs in a robot-aided auto-store logistic pipeline, showing its effectiveness and practicality.

show abstract

Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling

Cited by 4 publications

References 41 publications

Unseen Object Instance Segmentation with Fully Test-time RGB-D Embeddings Adaptation

Unseen Object Instance Segmentation with Fully Test-time RGB-D Embeddings Adaptation

DUQIM-Net: Probabilistic Object Hierarchy Representation for Multi-View Manipulation

On Improving Boundary Quality of Instance Segmentation in Cluttered and Chaotic Scenarios

Contact Info

Product

Resources

About