Force-based robot learning of pouring skills using parametric hidden Markov models

Rozo, Leonel; Jiménez, Pablo; Torras, Carme

doi:10.1109/romoco.2013.6614613

Cited by 60 publications

(48 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Robot Liquid Pouring. In the robotics community, there are a number of works [36,37,38,39,40,41,42] directly tackle the manipulating task of liquid pouring without considering the monitoring task. [36] build a liquid dynamic model using optical flow.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Liquid Pouring Monitoring via Rich Sensory Inputs

Tie-jun

Lin

Wang

et al. 2018

Computer Vision – ECCV 2018

View full text Add to dashboard Cite

Humans have the amazing ability to perform very subtle manipulation task using a closed-loop control system with imprecise mechanics (i.e., our body parts) but rich sensory information (e.g., vision, tactile, etc.). In the closed-loop system, the ability to monitor the state of the task via rich sensory information is important but often less studied. In this work, we take liquid pouring as a concrete example and aim at learning to continuously monitor whether liquid pouring is successful (e.g., no spilling) or not via rich sensory inputs. We mimic humans' rich sensories using synchronized observation from a chest-mounted camera and a wrist-mounted IMU sensor. Given many success and failure demonstrations of liquid pouring, we train a hierarchical LSTM with late fusion for monitoring. To improve the robustness of the system, we propose two auxiliary tasks during training: inferring (1) the initial state of containers and (2) forecasting the one-step future 3D trajectory of the hand with an adversarial training procedure. These tasks encourage our method to learn representation sensitive to container states and how objects are manipulated in 3D. With these novel components, our method achieves ∼ 8% and ∼ 11% better monitoring accuracy than the baseline method without auxiliary tasks on unseen containers and unseen users respectively.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Tamosiunaite et al [37] apply model-based reinforcement learning. Rozo et al [38] propose a parametric hidden Markov model to direct regress control commands. Brandl et al [39] learn to generalize pouring to unseen containers by warping the functional parts of the unseen containers to mimic the functional parts of a seen container.…”

Section: Related Workmentioning

confidence: 99%

Liquid Pouring Monitoring via Rich Sensory Inputs

Tie-jun

Lin

Wang

et al. 2018

Computer Vision – ECCV 2018

View full text Add to dashboard Cite

show abstract

“…Pouring is a challenging skill for a robot to learn (Rozo et al, 2013b). We want the robot to learn how to pour into a glass liquid from a bottle that can be filled up to different levels.…”

Section: Pouring Taskmentioning

confidence: 99%

Learning Controllers for Reactive and Proactive Behaviors in Human–Robot Collaboration

2016

Self Cite

View full text Add to dashboard Cite

Designed to safely share the same workspace as humans and assist them in various tasks, the new collaborative robots are targeting manufacturing and service applications that once were considered unattainable. The large diversity of tasks to carry out, the unstructured environments, and the close interaction with humans call for collaborative robots to seamlessly adapt their behaviors, so as to cooperate with the users successfully under different and possibly new situations (characterized, for example, by positions of objects/landmarks in the environment or by the user pose). This paper investigates how controllers capable of reactive and proactive behaviors in collaborative tasks can be learned from demonstrations. The proposed approach exploits the temporal coherence and dynamic characteristics of the task observed during the training phase to build a probabilistic model that enables the robot to both react to the user actions and lead the task when needed. The method is an extension of the hidden semi-Markov model where the duration probability distribution is adapted according to the interaction with the user. This adaptive duration hidden semi-Markov model (ADHSMM) is used to retrieve a sequence of states governing a trajectory optimization that provides the reference and gain matrices to the robot controller. A proof-of-concept evaluation is first carried out in a pouring task. The proposed framework is then tested in a collaborative task using a 7-DOF backdrivable manipulator.

show abstract

“…On the other hand, haptic sensing is another important modality for the perception of robotic pouring. For example, when the force and torque feedback on the manipulator is available, we can either estimate the volume of liquid being poured or directly learn a pouring policy in an end-to-end manner [6]. However, the correlation between haptic information and the pouring liquid can be rather complicated and are varied among different end effectors and containers.…”

Section: Introductionmentioning

confidence: 99%

“…Specifically, force data is exerted to generate pouring trajectories by predicting the angular velocity of the pouring container in simulation [21]. Rozo et al [6] used a parametric hidden Markov model to retrieve joint-level commands given the force-torque inputs from the human demonstration. Hannes et al [22] examined the viscosity estimation of the various liquids from tactile sensory data.…”

Section: Introductionmentioning

confidence: 99%

Making Sense of Audio Vibration for Liquid Height Estimation in Robotic Pouring

Liang

Ma³

et al. 2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

In this paper, we focus on the challenging perception problem in robotic pouring. Most of the existing approaches either leverage visual or haptic information. However, these techniques may suffer from poor generalization performances on opaque containers or concerning measuring precision. To tackle these drawbacks, we propose to make use of audio vibration sensing and design a deep neural network PouringNet to predict the liquid height from the audio fragment during the robotic pouring task. PouringNet is trained on our collected real-world pouring dataset with multimodal sensing data, which contains more than 3000 recordings of audio, force feedback, video and trajectory data of the human hand that performs the pouring task. Each record represents a complete pouring procedure. We conduct several evaluations on PouringNet with our dataset and robotic hardware. The results demonstrate that our PouringNet generalizes well across different liquid containers, positions of the audio receiver, initial liquid heights and types of liquid, and facilitates a more robust and accurate audio-based perception for robotic pouring.

show abstract

Force-based robot learning of pouring skills using parametric hidden Markov models

Cited by 60 publications

References 20 publications

Liquid Pouring Monitoring via Rich Sensory Inputs

Liquid Pouring Monitoring via Rich Sensory Inputs

Learning Controllers for Reactive and Proactive Behaviors in Human–Robot Collaboration

Making Sense of Audio Vibration for Liquid Height Estimation in Robotic Pouring

Contact Info

Product

Resources

About