Rearrangement with Nonprehensile Manipulation Using Deep Reinforcement Learning

Abstract

Rearranging objects on a tabletop surface by means of nonprehensile manipulation is a task which requires skillful interaction with the physical world. Usually, this is achieved by precisely modeling physical properties of the objects, robot, and the environment for explicit planning. In contrast, as explicitly modeling the physical environment is not always feasible and involves various uncertainties, we learn a nonprehensile rearrangement strategy with deep reinforcement learning based on only visual feedback. For this, we model the task with rewards and train a deep Q-network. Our potential field-based heuristic exploration strategy reduces the amount of collisions which lead to suboptimal outcomes and we actively balance the training set to avoid bias towards poor examples. Our training process leads to quicker learning and better performance on the task as compared to uniform exploration and standard experience replay. We demonstrate empirical evidence from simulation that our method leads to a success rate of 85%, show that our system can cope with sudden changes of the environment, and compare our performance with human level performance.

Review of Deep Reinforcement Learning-Based Object Grasping: Techniques, Open Challenges, and Recommendations

“…In terms of learning the nonprehensile rearrangement task in clutter, many feasible frameworks have been presented. For example, Yuan et al [137] learned nonprehensile rearrangement strategy based on Deep Q-network algorithm by exploiting heuristic exploration strategy for reducing the amount of collisions. Whereas, Monte-Carlo Tree Search exploration strategy, which relies on visual inputs coming from an RGB camera, has been presented to learn nonprehensile rearrangement task [138] [118].…”

A model predictive approach for online mobile manipulation of non-holonomic objects using learned dynamics

“…Li et al (2018) used a deep recurrent neural network model to learn object motion properties for planar pushing for a single object. Yuan et al (2018) designed a learning system that treats perception, action planning, and motion planning in an end-to-end process. Meriçli et al (2015) learned case-based planar motion of objects on the plane which is object-specific.…”

Reinforcement Learning in Topology-based Representation for Human Body Movement with Whole Arm Manipulation

“…Since deep reinforcement learning has shown success in complex artificial domains [25,26], controlling robots with reinforcement learning has become increasingly interesting [27]. For instance, it has been used to learn grasping [28,29] or manipulation in dynamic environments [5,30]. While these works exploit the advantages of deep models for visual input, this makes it difficult for them to generalize to different conditions.…”

“…Robotic manipulation is a complex problem that is often approached by grasping [1,2] or non-prehensile pushing [3][4][5]. However, when heavy or bulky objects need to be manipulated whole arm manipulation (WAM) is usually much more suitable [6][7][8][9].…”