Eye in hand: Towards GPU accelerated online grasp planning based on pointclouds from in-hand sensor

Hermann, A. M.; Mauch, Felix; Klemm, Sebastian; Roennau, Arne; Dillmann, Ruediger

doi:10.1109/humanoids.2016.7803394

Cited by 6 publications

(4 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The approach integrates both shape and appearance information into an articulated ICP approach to track the robot's manipulator and the object while improving the 3D model of the object. Similarly, another work [70] attaches a depth sensor to a robotic hand and plans grasps directly in the sensed voxel grid. These approaches improve their models of the object using only a single sensory modality but from multiple points in time.…”

Section: Robotic Visual Shape Understandingmentioning

confidence: 99%

Learning Mobile Manipulation

Watkins¹

2022

Preprint

View full text Add to dashboard Cite

Providing mobile robots with the ability to manipulate objects has, despite decades of research, remained a challenging problem. The problem is approachable in constrained environments where there is ample prior knowledge of the environment layout and manipulatable objects. The challenge is in building systems that scale beyond specific situational instances and gracefully operate in novel conditions. In the past, researchers used heuristic and simple rule-based strategies to accomplish tasks such as scene segmentation or reasoning about occlusion. These heuristic strategies work in constrained environments where a roboticist can make simplifying assumptions about everything from the geometries of the objects to be interacted with, level of clutter, camera position, lighting, and a myriad of other relevant variables. The work in this thesis will demonstrate how to build a system for robotic mobile manipulation that is robust to changes in these variables. This robustness will be enabled by recent simultaneous advances in the fields of big data, deep learning, and simulation. The ability of simulators to create realistic sensory data enables the generation of massive corpora of labeled training data for various grasping and navigation-based tasks. It is now possible to build systems that work in the real world trained using deep learning entirely on synthetic data. The ability to train and test on synthetic data allows for quick iterative development of new perception, planning and grasp execution algorithms that work in many environments.To build a robust system, this thesis introduces a novel multiple-view shape reconstruction architecture that leverages unregistered views of the object. To navigate to objects without localizing the agent, this thesis introduces a novel panoramic target goal architecture that takes previous views of the agent to inform a policy to navigate through an environment. Additionally, a novel next-best-view methodology is introduced to allow the agent to move around the object and refine its initial understanding of the object. The results show that this deep learned sim-to-real approach performs best when compared to heuristic-based methods in terms of reconstruction quality and success-weighted-by-path-length (SPL). This approach is also adaptable to the environment and robot chosen due to its modular design.I would like to acknowledge all members of the Columbia Robotics Lab (CRLab) for making my stay there an enjoyable and rewarding experience. In particular, I would like to thank

show abstract

Section: Robotic Visual Shape Understandingmentioning

confidence: 99%

Learning Mobile Manipulation

Watkins¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Other shape completion systems exist for household objects but not robotic grasping [25], [26]. Several geometric solutions to object 3D modeling have been proposed as well [27], [28], [29], [30].…”

Section: Related Workmentioning

confidence: 99%

Mobile Manipulation Leveraging Multiple Views

Watkins-Valls¹,

Allen²,

Maia³

et al. 2021

Preprint

View full text Add to dashboard Cite

While both navigation and manipulation are challenging topics in isolation, many tasks require the ability to both navigate and manipulate in concert. To this end, we propose a mobile manipulation system that leverages novel navigation and shape completion methods to manipulate an object with a mobile robot. Our system utilizes uncertainty in the initial estimation of a manipulation target to calculate a predicted next-best-view. Without the need of localization, the robot then uses the predicted panoramic view at the next-best-view location to navigate to the desired location, capture a second view of the object, create a new model that predicts the shape of object more accurately than a single image alone, and uses this model for grasp planning. We show that the system is highly effective for mobile manipulation tasks through simulation experiments using real world data, as well as ablations on each component of our system.

show abstract

“…The approach integrates both shape and appearance information into an articulated ICP approach to track the robot's manipulator and the object while improving the 3D model of the object. Similarly, another work [20] attaches a depth sensor to a robotic hand and plans grasps directly in the sensed voxel grid. These approaches improve their models of the object using only a single sensory modality but from multiple points in time.…”

Section: Related Workmentioning

confidence: 99%

Multi-Modal Geometric Learning for Grasping and Manipulation

Watkins-Valls

Varley

Allen

2019

2019 International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

This work provides an architecture that incorporates depth and tactile information to create rich and accurate 3D models useful for robotic manipulation tasks. This is accomplished through the use of a 3D convolutional neural network (CNN). Offline, the network is provided with both depth and tactile information and trained to predict the object's geometry, thus filling in regions of occlusion. At runtime, the network is provided a partial view of an object. Tactile information is acquired to augment the captured depth information. The network can then reason about the object's geometry by utilizing both the collected tactile and depth information. We demonstrate that even small amounts of additional tactile information can be incredibly helpful in reasoning about object geometry. This is particularly true when information from depth alone fails to produce an accurate geometric prediction. Our method is benchmarked against and outperforms other visual-tactile approaches to general geometric reasoning. We also provide experimental results comparing grasping success with our method.

show abstract

Eye in hand: Towards GPU accelerated online grasp planning based on pointclouds from in-hand sensor

Cited by 6 publications

References 12 publications

Learning Mobile Manipulation

Learning Mobile Manipulation

Mobile Manipulation Leveraging Multiple Views

Multi-Modal Geometric Learning for Grasping and Manipulation

Contact Info

Product

Resources

About