Deep reinforcement learning with smooth policy update: Application to robotic cloth manipulation

Tsurumine, Yoshihisa; Cui, Yunduan; Uchibe, Eiji; Matsubara, Takamitsu

doi:10.1016/j.robot.2018.11.004

Cited by 149 publications

(71 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Their method requires prior information about the shape of the object in order to reconstruct the missing market points. More recent work in using (deep) reinforcement learning for robotic folding also used vision-based methods to define the reward function for the agent [9,12]. However, relying solely on visual inputs and marker clues does not scale well.…”

Section: Related Workmentioning

confidence: 99%

“…Hand-engineered heuristics to estimate the cloth state have been researched [8] but are highly complex to reproduce and are prone to error. Visual cues such as colored surfaces or fiducial markers [11,12] can be attached to highly deformable materials but require the many occlusions caused by the self-collision of the cloth to be handled.…”

Section: Fitted Q-learningmentioning

confidence: 99%

“…A typical approach for the robotic folding of textile relies on the use of vision in order to detect grasping points and to perform texture segmentation and pose estimation [7,8,11], as well to estimate the state [9] and-in the case of reinforcement learning (RL)-based approaches-the reward function [12]. However, while vision is instrumental for recognizing and localizing objects, touch and force measurements become important once contact occurs, and the object is explored using the end-effector [13].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Simpler Learning of Robotic Manipulation of Clothing by Utilizing DIY Smart Textile Technology

et al. 2020

View full text Add to dashboard Cite

Deformable objects such as ropes, wires, and clothing are omnipresent in society and industry but are little researched in robotics research. This is due to the infinite amount of possible state configurations caused by the deformations of the deformable object. Engineered approaches try to cope with this by implementing highly complex operations in order to estimate the state of the deformable object. This complexity can be circumvented by utilizing learning-based approaches, such as reinforcement learning, which can deal with the intrinsic high-dimensional state space of deformable objects. However, the reward function in reinforcement learning needs to measure the state configuration of the highly deformable object. Vision-based reward functions are difficult to implement, given the high dimensionality of the state and complex dynamic behavior. In this work, we propose the consideration of concepts beyond vision and incorporate other modalities which can be extracted from deformable objects. By integrating tactile sensor cells into a textile piece, proprioceptive capabilities are gained that are valuable as they provide a reward function to a reinforcement learning agent. We demonstrate on a low-cost dual robotic arm setup that a physical agent can learn on a single CPU core to fold a rectangular patch of textile in the real world based on a learned reward function from tactile information.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Fitted Q-learningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Simpler Learning of Robotic Manipulation of Clothing by Utilizing DIY Smart Textile Technology

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Emphasis was placed on the learning of the robotic arms, and deep reinforcement learning was investigated to learn complex policies through high-level observations such as typing. Because deep reinforcement learning requires a large number of training samples, a method that improves the sample efficiency and learning stability with fewer samples by combining the characteristics of smooth policy updates with automatic feature extraction of deep neural networks was proposed [21]. Similarly, there has been a study involving learning robots that pick up and classify objects.…”

Section: Rl X Systemmentioning

confidence: 99%

Cooperative Multi-Agent Interaction and Evaluation Framework Considering Competitive Networks with Dynamic Topology Changes

Kim

Lee

2020

Applied Sciences

View full text Add to dashboard Cite

In recent years, the problem of reinforcement learning has become increasingly complex, and the computational demands with respect to such processes have increased. Accordingly, various methods for effective learning have been proposed. With the help of humans, the learning object can learn more accurately and quickly to maximize the reward. However, the rewards calculated by the system and via human intervention (that make up the learning environment) differ and must be used accordingly. In this paper, we propose a framework for learning the problems of competitive network topologies, wherein the environment dynamically changes agent, by computing the rewards via the system and via human evaluation. The proposed method is adaptively updated with the rewards calculated via human evaluation, making it more stable and reducing the penalty incurred while learning. It also ensures learning accuracy, including rewards generated from complex network topology consisting of multiple agents. The proposed framework contributes to fast training process using multi-agent cooperation. By implementing these methods as software programs, this study performs numerical analysis to demonstrate the effectiveness of the adaptive evaluation framework applied to the competitive network problem depicting the dynamic environmental topology changes proposed herein. As per the numerical experiments, the greater is the human intervention, the better is the learning performance with the proposed framework.

show abstract

“…It is experimentally shown that the dueling network architecture converges faster than the conventional single-stream network architecture (Wang et al, 2016;Tsurumine et al, 2019). Another important technological advance is entropy-based regularization (Ziebart et al, 2008;Mnih et al, 2016) that has been shown to improve both exploration and robustness, by adding the entropy of policy to the reward function.…”

Section: Techniques Together With Dqnmentioning

confidence: 99%

Constrained Deep Q-Learning Gradually Approaching Ordinary Q-Learning

Ohnishi

Uchibe²,

Yamaguchi

et al. 2019

Front. Neurorobot.

Self Cite

View full text Add to dashboard Cite

A deep Q network (DQN) (Mnih et al., 2013) is an extension of Q learning, which is a typical deep reinforcement learning method. In DQN, a Q function expresses all action values under all states, and it is approximated using a convolutional neural network. Using the approximated Q function, an optimal policy can be derived. In DQN, a target network, which calculates a target value and is updated by the Q function at regular intervals, is introduced to stabilize the learning process. A less frequent updates of the target network would result in a more stable learning process. However, because the target value is not propagated unless the target network is updated, DQN usually requires a large number of samples. In this study, we proposed Constrained DQN that uses the difference between the outputs of the Q function and the target network as a constraint on the target value. Constrained DQN updates parameters conservatively when the difference between the outputs of the Q function and the target network is large, and it updates them aggressively when this difference is small. In the proposed method, as learning progresses, the number of times that the constraints are activated decreases. Consequently, the update method gradually approaches conventional Q learning. We found that Constrained DQN converges with a smaller training dataset than in the case of DQN and that it is robust against changes in the update frequency of the target network and settings of a certain parameter of the optimizer. Although Constrained DQN alone does not show better performance in comparison to integrated approaches nor distributed methods, experimental results show that Constrained DQN can be used as an additional components to those methods.

show abstract

Deep reinforcement learning with smooth policy update: Application to robotic cloth manipulation

Cited by 149 publications

References 17 publications

Simpler Learning of Robotic Manipulation of Clothing by Utilizing DIY Smart Textile Technology

Simpler Learning of Robotic Manipulation of Clothing by Utilizing DIY Smart Textile Technology

Cooperative Multi-Agent Interaction and Evaluation Framework Considering Competitive Networks with Dynamic Topology Changes

Constrained Deep Q-Learning Gradually Approaching Ordinary Q-Learning

Contact Info

Product

Resources

About