Context-Aware Grasp Generation in Cluttered Scenes

Hoang, Dinh-Cuong; Stork, Johannes A.; Stoyanov, Todor

doi:10.1109/icra46639.2022.9811371

Cited by 15 publications

(13 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Object pose estimation, known as determining an object's precise position and orientation in 3D space from visual data, holds immense significance across various fields [1]- [10]. This capability is vital for machines to interact intelligently with the physical world.…”

Section: Introductionmentioning

confidence: 99%

Object Pose Estimation Using Color Images and Predicted Depth Maps

Hoang,

Xuan Tan,

Nguyen

et al. 2024

IEEE Access

Self Cite

View full text Add to dashboard Cite

The task of object pose estimation in computer vision heavily relies on both color (RGB) and depth (D) images to provide crucial appearance and geometric information, assisting algorithms in understanding occlusions and object geometry, thereby enhancing accuracy. However, the dependency on specialized sensors capable of capturing depth poses challenges in terms of cost and availability. Consequently, researchers are exploring methods to estimate object poses solely from RGB images. Nevertheless, this approach encounters difficulties in handling occlusions, discerning object geometry, and resolving ambiguities arising from similar color or texture patterns. This paper introduces a novel geometry-aware method for object pose estimation utilizing RGB images as input to determine the poses of multiple object instances. Our approach leverages both depth and color images during training but only relies on color images during inference. Departing from traditional depth sensors, our method computes predicted point clouds directly from estimated depth images derived from RGB inputs. A key innovation lies in the formulation of a multi-scale fusion module adept at seamlessly integrating features extracted from RGB images with those inferred from the predicted point clouds. This fusion process significantly fortifies the pose estimation pipeline by harnessing the strengths of both modalities, resulting in notably improved object poses. Extensive experimentation demonstrates that our approach markedly outperforms state-of-the-art RGB-based methods on Occluded-LINEMOD and YCB-Video datasets. Moreover, our method achieves competitive results compared to RGB-D approaches that necessitate both RGB and depth data from physical sensors.

show abstract

Section: Introductionmentioning

confidence: 99%

Object Pose Estimation Using Color Images and Predicted Depth Maps

Hoang,

Xuan Tan,

Nguyen

et al. 2024

IEEE Access

Self Cite

View full text Add to dashboard Cite

show abstract

“…An alternative approach to grasp configuration generation, also referred to as grasp detection, involves deriving grasp configurations directly from sensor data, without presum-ing knowledge of the object's 3D model or relying on precomputed grasps. This methodology is commonly referred to as grasp generation or grasp detection [2], [3], [7]. Current methods within this domain can be broadly categorized into two groups: planar grasping and six Degrees of Freedom (6-DoF) grasping.…”

Section: Introductionmentioning

confidence: 99%

“…This low degree of freedom (DoF) representation simplifies the task into a detection problem but may limit performance in more complex 3D manipulation tasks. On the other hand, 6-DoF grasping provides greater dexterity and is more suitable for handling intricate scenarios [2], [10]. However, the accurate generation of 6-DoF grasps often requires geometric information, leading many existing methods to depend on 3D point cloud data.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Attention-Based Grasp Detection With Monocular Depth Estimation

Xuan Tan,

Hoang,

Nguyen

et al. 2024

IEEE Access

Self Cite

View full text Add to dashboard Cite

Grasp detection plays a pivotal role in robotic manipulation, allowing robots to interact with and manipulate objects in their surroundings. Traditionally, this has relied on three-dimensional (3D) point cloud data acquired from specialized depth cameras. However, the limited availability of such sensors in real-world scenarios poses a significant challenge. In many practical applications, robots operate in diverse environments where obtaining high-quality 3D point cloud data may be impractical or impossible. This paper introduces an innovative approach to grasp generation using color images, thereby eliminating the need for dedicated depth sensors. Our method capitalizes on advanced deep learning techniques for depth estimation directly from color images. Instead of relying on conventional depth sensors, our approach computes predicted point clouds based on estimated depth images derived directly from Red-Green-Blue (RGB) input data. To our knowledge, this is the first study to explore the use of predicted depth data for grasp detection, moving away from the traditional dependence on depth sensors. The novelty of this work is the development of a fusion module that seamlessly integrates features extracted from RGB images with those inferred from the predicted point clouds. Additionally, we adapt a voting mechanism from our previous work (VoteGrasp) to enhance robustness to occlusion and generate collision-free grasps. Experimental evaluations conducted on standard datasets validate the effectiveness of our approach, demonstrating its superior performance in generating grasp configurations compared to existing methods. With our proposed method, we achieved a significant 4% improvement in average precision compared to state-of-the-art grasp detection methods. Furthermore, our method demonstrates promising practical viability through real robot grasping experiments, achieving an impressive 84% success rate.

show abstract

“…State-of-the-art robotic grasping systems [1], [2], [3] function well in moderately cluttered scenes, but are fundamentally limited in assuming that objects are directly graspable -that is, that there always exists a collisionfree grasp configuration within the reachability space of the robot arm. In practice, this assumption is often violated: for example, in cases when objects are tightly packed together, or placed in configurations that obstruct all feasible grasps (e.g., think of a book lying flat on a table).…”

Section: Introductionmentioning

confidence: 99%

Variable Impedance Skill Learning for Contact-Rich Manipulation

Yang

Dürr

Topp

et al. 2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Contact-rich manipulation tasks remain a hard problem in robotics that requires interaction with unstructured environments. Reinforcement Learning (RL) is one potential solution to such problems, as it has been successfully demonstrated on complex continuous control tasks. Nevertheless, current state-ofthe-art methods require policy training in simulation to prevent undesired behavior and later domain transfer even for simple skills involving contact. In this paper, we address the problem of learning contact-rich manipulation policies by extending an existing skill-based RL framework with a variable impedance action space. Our method leverages a small set of suboptimal demonstration trajectories and learns from both position, but also crucially impedance-space information. We evaluate our method on a number of peg-in-hole task variants with a Franka Panda arm and demonstrate that learning variable impedance actions for RL in Cartesian space can be deployed directly on the real robot, without resorting to learning in simulation.

show abstract

Context-Aware Grasp Generation in Cluttered Scenes

Cited by 15 publications

References 36 publications

Object Pose Estimation Using Color Images and Predicted Depth Maps

Object Pose Estimation Using Color Images and Predicted Depth Maps

Attention-Based Grasp Detection With Monocular Depth Estimation

Variable Impedance Skill Learning for Contact-Rich Manipulation

Contact Info

Product

Resources

About