Trends and challenges in robot manipulation

Billard, Aude; Kragić, Danica

doi:10.1126/science.aat8414

Cited by 528 publications

(277 citation statements)

References 99 publications

Supporting

Mentioning

236

Contrasting

Order By: Relevance

“…This paper 1 makes multiple contributions that advance event-driven visual-tactile perception. First, to enable richer tactile sensing, we contribute the 39-taxel NeuTouch fingertip sensor.…”

Section: Introductionmentioning

confidence: 99%

Event-Driven Visual-Tactile Sensing and Learning for Robots

Taunyazov¹,

Sng²,

Lim³

et al. 2020

Robotics: Science and Systems XVI

View full text Add to dashboard Cite

This work contributes an event-driven visual-tactile perception system, comprising a novel biologically-inspired tactile sensor and multi-modal spike-based learning. Our neuromorphic fingertip tactile sensor, NeuTouch, scales well with the number of taxels thanks to its event-based nature. Likewise, our Visual-Tactile Spiking Neural Network (VT-SNN) enables fast perception when coupled with event sensors. We evaluate our visual-tactile system (using the NeuTouch and Prophesee event camera) on two robot tasks: container classification and rotational slip detection. On both tasks, we observe good accuracies relative to standard deep learning methods. We have made our visual-tactile datasets freely-available to encourage research on multi-modal event-driven robot perception, which we believe is a promising approach towards intelligent power-efficient robot systems.

show abstract

Section: Introductionmentioning

confidence: 99%

Event-Driven Visual-Tactile Sensing and Learning for Robots

Taunyazov¹,

Sng²,

Lim³

et al. 2020

Robotics: Science and Systems XVI

View full text Add to dashboard Cite

show abstract

“…Robotic manipulation work tends to focus on rigid objects (Bohg et al, 2014;Billard and Kragic, 2019). However, most objects manipulated by animals and humans change shape upon contact.…”

Section: Introductionmentioning

confidence: 99%

Modeling of Deformable Objects for Robotic Manipulation: A Tutorial and Review

et al. 2020

Self Cite

View full text Add to dashboard Cite

Manipulation of deformable objects has given rise to an important set of open problems in the field of robotics. Application areas include robotic surgery, household robotics, manufacturing, logistics, and agriculture, to name a few. Related research problems span modeling and estimation of an object's shape, estimation of an object's material properties, such as elasticity and plasticity, object tracking and state estimation during manipulation, and manipulation planning and control. In this survey article, we start by providing a tutorial on foundational aspects of models of shape and shape dynamics. We then use this as the basis for a review of existing work on learning and estimation of these models and on motion planning and control to achieve desired deformations. We also discuss potential future lines of work.

show abstract

“…A typical approach for the robotic folding of textile relies on the use of vision in order to detect grasping points and to perform texture segmentation and pose estimation [7,8,11], as well to estimate the state [9] and-in the case of reinforcement learning (RL)-based approaches-the reward function [12]. However, while vision is instrumental for recognizing and localizing objects, touch and force measurements become important once contact occurs, and the object is explored using the end-effector [13]. Recent work [14] has illustrated this idea by applying a touch-based control method for a ball repositioning task, rolling a dice and the deflection of a stick.…”

Section: Introductionmentioning

confidence: 99%

Simpler Learning of Robotic Manipulation of Clothing by Utilizing DIY Smart Textile Technology

et al. 2020

View full text Add to dashboard Cite

Deformable objects such as ropes, wires, and clothing are omnipresent in society and industry but are little researched in robotics research. This is due to the infinite amount of possible state configurations caused by the deformations of the deformable object. Engineered approaches try to cope with this by implementing highly complex operations in order to estimate the state of the deformable object. This complexity can be circumvented by utilizing learning-based approaches, such as reinforcement learning, which can deal with the intrinsic high-dimensional state space of deformable objects. However, the reward function in reinforcement learning needs to measure the state configuration of the highly deformable object. Vision-based reward functions are difficult to implement, given the high dimensionality of the state and complex dynamic behavior. In this work, we propose the consideration of concepts beyond vision and incorporate other modalities which can be extracted from deformable objects. By integrating tactile sensor cells into a textile piece, proprioceptive capabilities are gained that are valuable as they provide a reward function to a reinforcement learning agent. We demonstrate on a low-cost dual robotic arm setup that a physical agent can learn on a single CPU core to fold a rectangular patch of textile in the real world based on a learned reward function from tactile information.

show abstract

Trends and challenges in robot manipulation

Cited by 528 publications

References 99 publications

Event-Driven Visual-Tactile Sensing and Learning for Robots

Event-Driven Visual-Tactile Sensing and Learning for Robots

Modeling of Deformable Objects for Robotic Manipulation: A Tutorial and Review

Simpler Learning of Robotic Manipulation of Clothing by Utilizing DIY Smart Textile Technology

Contact Info

Product

Resources

About