Visual Repetition Sampling for Robot Manipulation Planning

Puang, En Yen; Lehner, Peter; Márton, Zoltán-Csaba; Durner, Maximilian; Triebel, Rudolph; Albu‐Schäffer, Alin

doi:10.1109/icra.2019.8793942

Cited by 8 publications

(5 citation statements)

References 22 publications

(27 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Autoencoders and their variations have also been applied in many robotic applications to deal with perception in manipulation [13], [14]. An autoencoder maps the input image to a latent space, where the encoded latent value is usually used for planning [15] or as a feature extraction component for further learning process [14].…”

Section: A Visual Perception and Representation For Manipulationmentioning

confidence: 99%

“…In addition, adversarial examples are used to effectively augment training data which the hand-crafted methods could not achieve [27]. A randomized combination of multiple generation methods including Fast Gradient Sign Method (FGSM), iterative FGSM and least-likely FGSM are used together with random augmentation strength and number of iterations as in [14]. These augmentations are applied to the entire mini-batch before being used for training.…”

Section: End-to-end Training and Zero-shot Transfermentioning

confidence: 99%

“…The decoder on the other hand is a feedforward expanding DenseNets while the servo is made of multiple Fully-Connected layers. All DenseNet used here are of Weight-BatchNorm-Activation form and all convolutional layers has increasing dilation as in [14]. The total learnable parameters of KOVIS is around 1M.…”

Section: Implementations and Experimentsmentioning

confidence: 99%

See 2 more Smart Citations

KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation

Puang,

Tee,

Jing

2020

Preprint

Self Cite

View full text Add to dashboard Cite

We present KOVIS, a novel learning-based, calibration-free visual servoing method for fine robotic manipulation tasks with eye-in-hand stereo camera system. We train the deep neural network only in the simulated environment; and the trained model could be directly used for real-world visual servoing tasks. KOVIS consists of two networks. The first keypoint network learns the keypoint representation from the image using with an autoencoder. Then the visual servoing network learns the motion based on keypoints extracted from the camera image. The two networks are trained end-to-end in the simulated environment by self-supervised learning without manual data labeling. After training with data augmentation, domain randomization, and adversarial examples, we are able to achieve zero-shot sim-to-real transfer to real-world robotic manipulation tasks. We demonstrate the effectiveness of the proposed method in both simulated environment and realworld experiment with different robotic manipulation tasks, including grasping, peg-in-hole insertion with 4mm clearance, and M13 screw insertion. The demo video

show abstract

Section: A Visual Perception and Representation For Manipulationmentioning

confidence: 99%

Section: End-to-end Training and Zero-shot Transfermentioning

confidence: 99%

See 1 more Smart Citation

KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation

Puang,

Tee,

Jing

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, it requires a good object detector and a pose estimator, which also separate the pipeline into two stages. One other approach learns a low dimensional latent space representation of the input image with an autoencoder [25], [26], [27]. However, these latent space features usually encode the entire physical world in the camera view which limits its generalizability.…”

Section: B Representation Learning For Visual Inputsmentioning

confidence: 99%

End-to-end Reinforcement Learning of Robotic Manipulation with Robust Keypoints Representation

Wang¹,

Puang²,

Lee³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

We present an end-to-end Reinforcement Learning (RL) framework for robotic manipulation tasks, using a robust and efficient keypoints representation. The proposed method learns keypoints from camera images as the state representation, through a self-supervised autoencoder architecture. The keypoints encode the geometric information, as well as the relationship of the tool and target in a compact representation to ensure efficient and robust learning. After keypoints learning, the RL step then learns the robot motion from the extracted keypoints state representation. The keypoints and RL learning processes are entirely done in the simulated environment. We demonstrate the effectiveness of the proposed method on robotic manipulation tasks including grasping and pushing, in different scenarios. We also investigate the generalization capability of the trained model. In addition to the robust keypoints representation, we further apply domain randomization and adversarial training examples to achieve zero-shot sim-to-real transfer in real-world robotic manipulation tasks.

show abstract

“…Semantic segmentation is a fundamental task for various applications such as autonomous driving [1,2], biomedical image analysis [3,4,5], remote sensing [6] and robot manipulation [7]. Recently, data-driven methods have achieved great success with large-scale datasets [8,9].…”

Section: Introductionmentioning

confidence: 99%

DEAL: Difficulty-aware Active Learning for Semantic Segmentation

Xie

Feng

Chen

et al. 2020

Preprint

View full text Add to dashboard Cite

Active learning aims to address the paucity of labeled data by finding the most informative samples. However, when applying to semantic segmentation, existing methods ignore the segmentation difficulty of different semantic areas, which leads to poor performance on those hard semantic areas such as tiny or slender objects. To deal with this problem, we propose a semantic Difficulty-awarE Active Learning (DEAL) network composed of two branches: the common segmentation branch and the semantic difficulty branch. For the latter branch, with the supervision of segmentation error between the segmentation result and GT, a pixel-wise probability attention module is introduced to learn the semantic difficulty scores for different semantic areas. Finally, two acquisition functions are devised to select the most valuable samples with semantic difficulty. Competitive results on semantic segmentation benchmarks demonstrate that DEAL achieves state-of-the-art active learning performance and improves the performance of the hard semantic areas in particular.

show abstract

Visual Repetition Sampling for Robot Manipulation Planning

Cited by 8 publications

References 22 publications

KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation

KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation

End-to-end Reinforcement Learning of Robotic Manipulation with Robust Keypoints Representation

DEAL: Difficulty-aware Active Learning for Semantic Segmentation

Contact Info

Product

Resources

About