KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation

Abstract

We present KOVIS, a novel learning-based, calibration-free visual servoing method for fine robotic manipulation tasks with eye-in-hand stereo camera system. We train the deep neural network only in the simulated environment; and the trained model could be directly used for real-world visual servoing tasks. KOVIS consists of two networks. The first keypoint network learns the keypoint representation from the image using with an autoencoder. Then the visual servoing network learns the motion based on keypoints extracted from the camera image. The two networks are trained end-to-end in the simulated environment by self-supervised learning without manual data labeling. After training with data augmentation, domain randomization, and adversarial examples, we are able to achieve zero-shot sim-to-real transfer to real-world robotic manipulation tasks. We demonstrate the effectiveness of the proposed method in both simulated environment and realworld experiment with different robotic manipulation tasks, including grasping, peg-in-hole insertion with 4mm clearance, and M13 screw insertion. The demo video

Coarse-to-Fine for Sim-to-Real: Sub-Millimetre Precision Across Wide Task Spaces

“…Closer to our work, Puang et al [20], show that it is possible to be precise with more general, end-to-end simto-real policies. Their work however also focuses on lastinch manipulation only, ignoring the issue of how to incorporate such methods to longer horizon tasks.…”

“…modalities available from cameras, namely depth [13], stereo IR (often available on depth cameras) [20], and RGB [10], but it is unclear which would work better in an eye-in-hand sim-to-real setting with sub-mm precision requirements. As another example, image keypoints as a feature representation have been shown to work well [20], and conceptually provide a promising, well regularised representation which should facilitate domain-invariance for sim-to-real transfer. Nonetheless, to the best of our knowledge, this has not yet been verified by comparing keypoints directly to more standard feature representations.…”

“…2). Past sim-to-real works have chosen between those in order to construct their methods [10], [13], [20], but it is unclear to the best of our knowledge if there is any merit in choosing any one of those modalities over the other. In this experiment, we answer this question by benchmarking the following input options: (1) RGB, (2) stereo IR, (3) depth, and (4) grayscale.…”

“…Deciding on a Feature Representation 1) Experimental Procedure: In order for an image-based neural network to transfer well between a simulator and the real world, it is important that its learned features are invariant to the visual discrepancies between the two domains. Traditionally, domain randomisation with simple convolutional features have been used [10], and recently encoderdecoder structures with keypoint representations have also shown good results [20]. Conceptually, the latter should provide strong regularisation, helping the network focus on the geometric aspects of the image, especially when coupled to a depth reconstruction auxiliary loss.…”

“…This can be addressed by using a simulator and zeroshot sim-to-real transfer [10], [20], [30], eliminating the need for any real-world data at all. However, to the best of our knowledge, existing methods in the field either achieve high precision for tasks where the object remains in a very small task space [7], or they achieve generalisation across the workspace but do not consider highly precise control [10].…”