Affordance Learning for End-to-End Visuomotor Robot Control

Hämäläinen, Aleksi; Arndt, Karol; Ghadirzadeh, Ali; Kyrki, Ville

doi:10.1109/iros40897.2019.8968596

Cited by 36 publications

(36 citation statements)

References 26 publications

(44 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…James et al trained a visuomotor skill for pick-and-place task in simulation with domain randomization and transferred the visuomotor skill to the real world without any fine-tuning [3]. Hämäläinen et al applied an affordance detection method to compress task-related features and trained a visuomotor policy with these features to get a good generalization capability to new tasks and new objects [23]. Chen et al used an adversarial feature to enhance the robustness of the end-to-end visuomotor skills, which is integrated with RL framework [24].…”

Section: Related Workmentioning

confidence: 99%

Object Detection-Based One-Shot Imitation Learning with an RGB-D Camera

Shao

et al. 2020

Applied Sciences

View full text Add to dashboard Cite

End-to-end robot learning has achieved a great success for robots to obtain various manipulation skills. It learns a function which maps visual information to robotic action directly. Because of the diversity of target objects, most end-to-end robot learning approaches have focused on a single object-specific task with a limited capability of generalization. In this work, an object detection-based one-shot learning method is proposed, which separates the semantic understanding from robot control. It enables a robot to acquire similar manipulation skills efficiently and to have the ability to cope with new objects with a single demonstration. This approach mainly has two modules: the object detection network and the motion policy network. With RGB images, the object detection network tries to output the task-related semantic keypoint of the target object, which is the center of the container in this application, and the motion policy network generates the motion action based on the depth map and the detected keypoint. To evaluate this proposed pipeline, a series of experiments are conducted on typical placing tasks in different simulation scenarios and, additionally, the learned policy is transferred from simulation to the real world without any fine-tuning.

show abstract

Section: Related Workmentioning

confidence: 99%

Object Detection-Based One-Shot Imitation Learning with an RGB-D Camera

Shao

et al. 2020

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…They could not be applied on robot manipulation task directly. A trajectory latent space was also used for motion generation of robot manipulation, which embedded the entire task-specific trajectory into a low dimensional latent space [40], [41]. It used one RGB image to generate the entire manipulation trajectory just like our framework.…”

Section: Related Workmentioning

confidence: 99%

Location Instruction-Based Motion Generation for Sequential Robotic Manipulation

Shao

Wang

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Deep learning-based visuomotor control for manipulation are studied recently, which uses a neural network to learn the mapping from images to robotic actions directly. Most previous methods assume there is only one target object in the image. When multiple target objects are present, they are normally regarded as multi-task problems. One-hot vectors are often used to denote different tasks in the visuomotor framework. Nonetheless, these one-hot vector-based methods are easily disturbed and non-extendable. In this paper, a location instruction is used to guide the robot to generate different trajectories in the situation with multiple targets. The proposed framework mainly consists of three modules: the AutoEncoder (AE) network, the motion generation network and the location detection network. AE tries to process the perception information into small-scale feature maps. The location detection network gives the pixel coordinates of the center of the target object in the image. The motion generation network combines the location instruction and the preprocessed perception information and generates an entire motion trajectory to finish the specified manipulation task. For the object stacking tasks with distractors, the proposed method could obtain a success rate of 98%, while the one-hot vector-based method only has a success rate of 56%.

show abstract

“…Compression processing is also widely used for redundant high-dimensional data such as images or video frames. Variational autoencoder [47] is often used to compress the perceptional information in field of robot vision [48][49][50]. It is verified that VAE has better generalization ability compared with traditional autoencoder (AE) in visuomotor framework [51].…”

Section: Perception Networkmentioning

confidence: 99%

“…Hoof et al used VAE to process the high-dimensional tactile and visual feedback for stabilization task [48]. Hämäläinen et al applied VAE to cope with the visual observation for affordance learning in a visuomotor control framework [49]. Piergiovanni compressed the image via a VAE for robot navigation [50].…”

Section: Perception Networkmentioning

confidence: 99%

Composable Instructions and Prospection Guided Visuomotor Control for Robotic Manipulation

Shao¹,

Hu²,

Wang³

et al. 2019

IJCIS

View full text Add to dashboard Cite

Deep neural network-based end-to-end visuomotor control for robotic manipulation is becoming a hot issue of robotics field recently. One-hot vector is often used for multi-task situation in this framework. However, it is inflexible using one-hot vector to describe multiple tasks and transmit intentions of humans. This paper proposes a framework by combining composable instructions with visuomotor control for multi-task problems. The framework mainly consists of two modules: variational autoencoder (VAE) networks and long short-term memory (LSTM) networks. Perception information of the environment is encoded by VAE into a small latent space. The embedded perception information and composable instructions are combined by the LSTM module to guide robotic motion based on different intentions. Prospection is also used to learn the purposes of instructions, which means not only predicting the next action but also predicting a sequence of future actions at the same time. To evaluate this framework, a series of experiments are conducted in pick-and-place application scenarios. For new tasks, the framework could obtain a success rate of 91.2%, which means it has a good generalization ability.

show abstract

Affordance Learning for End-to-End Visuomotor Robot Control

Cited by 36 publications

References 26 publications

Object Detection-Based One-Shot Imitation Learning with an RGB-D Camera

Object Detection-Based One-Shot Imitation Learning with an RGB-D Camera

Location Instruction-Based Motion Generation for Sequential Robotic Manipulation

Composable Instructions and Prospection Guided Visuomotor Control for Robotic Manipulation

Contact Info

Product

Resources

About