Object-Independent Human-to-Robot Handovers using Real Time Robotic Vision

Abstract

We present an approach for safe and objectindependent human-to-robot handovers using real time robotic vision and manipulation. We aim for general applicability with a generic object detector, a fast grasp selection algorithm and by using a single gripper-mounted RGB-D camera, hence not relying on external sensors. The robot is controlled via visual servoing towards the object of interest. Putting a high emphasis on safety, we use two perception modules: human body part segmentation and hand/finger segmentation. Pixels that are deemed to belong to the human are filtered out from candidate grasp poses, hence ensuring that the robot safely picks the object without colliding with the human partner. The grasp selection and perception modules run concurrently in real-time, which allows monitoring of the progress. In experiments with 13 objects, the robot was able to successfully take the object from the human in 81.9% of the trials.

Gesture Recognition for Initiating Human-to-Robot Handovers

“…Human body keypoints is commonly used in robotics applications, such as for ensuring safety for Human-Robot Interaction [6], [12] and recognizing gestures such as a pointing gestures [13].…”

“…Human-to-Robot handovers have been demonstrated on a physical robot system by a handful of researchers [1]- [6]. Even though successful demonstrations has been shown, in these works the robot is programmed for a single task: object handovers.…”

Reactive Human-to-Robot Handovers of Arbitrary Objects

“…Among the surveyed papers that involved real human-robot interactions, the majority focused on robot-tohuman (R2H) handovers, while 11 involved human-to-robot (H2R) handovers. Most papers (26 out of 38) involved a single object being handed over; only one paper had more than six objects (i.e., Rosenberg et al with 13 objects [5]). Hence, despite the great progress made in the field, there is a gap in generalization to larger sets of objects, or ultimately, arbitrary objects.…”

“…Konstantinova et al aim to address the challenge of handing arbitrary objects, based on a method that relies only on wrist force sensors (i.e., no vision or tactile information), but still requires the person to bring the object in contact with the robot gripper [18]. Most closely related to this paper is recent work by Rosenberg et al who developed a method for grasping objects that can be recognized by the robot from a human giver's hand [5]. Their object detector is based on the YOLO V3 object detector [19] trained on 80 object categories from the COCO [20] dataset and they generate grasps using a modified GG-CNN [21].…”

“…Their approach was evaluated with 13 diverse objects handed in different configurations. The work presented here has two key differences: (1) our object detection and segmentation does not require objects to be in the pre-trained object dataset, which is the main bottleneck discussed in [5], hence can handle truly arbitrary objects, and (2) our system is closed-loop and the grasp selection is refined over time to enable reactivity by taking advantage of the robust tracking of the segmented hand and object.…”