Dynamics Learning With Object-Centric Interaction Networks for Robot Manipulation

Abstract

Understanding the physical interactions of objects with environments is critical for multiobject robotic manipulation tasks. A predictive dynamics model can predict the future states of manipulated objects, which is used to plan plausible actions that enable the objects to achieve desired goal states. However, most current approaches on dynamics learning from high-dimensional visual observations have limitations. These methods either rely on a large amount of real-world data or build a model with a fixed number of objects, which makes them difficult to generalize to unseen objects. This paper proposes a Deep Object-centric Interaction Network (DOIN) which encodes object-centric representations for multiple objects from raw RGB images and reasons about the future trajectory for each object in latent space. The proposed model is trained only on large amounts of random interaction data collected in simulation. The learned model combined with a model predictive control framework enables a robot to search action sequences that manipulate objects to the desired configurations. The proposed method is evaluated both in simulation and real-world experiments on multi-object pushing tasks. Extensive simulation experiments show that DOIN can achieve high prediction accuracy in different scenes with different numbers of objects and outperform state-of-the-art baselines in the manipulation tasks. Real-world experiments demonstrate that the model trained on simulated data can be transferred to the real robot and can successfully perform multi-object pushing tasks for previously-unseen objects with significant variations in shape and size. INDEX TERMSDeep Learning in robotic manipulation, model learning, representation learning, visual Learning. Recently, video prediction models have been successfully applied to perform various manipulation tasks [9]-[11],

Review of Learning-Based Robotic Manipulation in Cluttered Environments

“…Although learning from a curriculum of increasingly difficult long-horizon tasks appears to be a natural solution, it does not work in many situations [ 84 ]. As pointed out by [ 121 ], such approaches include only simple situations, such as falling blocks, and do not contain behaviours relevant to robotic manipulation tasks. Currently, the curriculum is being produced manually and is based on the premise that smaller groupings of objects have easier learning and control capabilities than larger groups of objects.…”

Development of Multi-Robotic Arm System for Sorting System Using Computer Vision

“…The robotic system integrated with a vision system has enabled robots to accomplish many applications as assembly and disassembly, vision guide robot, mapping, navigation, tracking, path planning, robot localization, exploration, surveillance, search, recognition, inspection [12][13][14][15][16][17][18][19][20].…”