Detecting Features of Tools, Objects, and Actions from Effects in a Robot using Deep Learning

Abstract

We propose a tool-use model that can detect the features of tools, target objects, and actions from the provided effects of object manipulation. We construct a model that enables robots to manipulate objects with tools, using infant learning as a concept. To realize this, we train sensory-motor data recorded during a tool-use task performed by a robot with deep learning. Experiments include four factors: (1) tools, (2) objects, (3) actions, and (4) effects, which the model considers simultaneously. For evaluation, the robot generates predicted images and motions given information of the effects of using unknown tools and objects. We confirm that the robot is capable of detecting features of tools, objects, and actions by learning the effects and executing the task.

Tool-Use Model to Reproduce the Goal Situations Considering Relationship Among Tools, Objects, Actions and Effects Using Multimodal Deep Neural Networks

“…In the previous study Ref. Saito et al (2018a) , the adopted sensor was only image without force, so they could not use small objects and had many restrictions on the types of actions, because if an object was occluded by the arm, it would be misrecognized. Thanks to these achievement, our model can contribute to the realization of robots that can handle various tasks, it can greatly impact practical applications for robots in everyday environments.…”

“…Cf nodes with small time constants learn movement primitives in the data, whereas Cs nodes with large time constants learn sequences. By combining these three node types, long, complex time series data can be learned, the usefulness of which for manipulation has been confirmed in several studies Yang et al (2016) , Takahashi et al (2017) , and Saito et al (2018b , a , 2020 , 2021) .…”

“…Figure 1 shows the way to control the robot using the model, which is extended from Ref. Saito et al (2018a) to deal with both image and force sensor data efficiently. The tool-use model comprises two modules, a feature extraction module and a motion generation module.…”

Example-Driven Trajectory Learner for Robots under Structured Static Environment

“…In recent years, task learning in robots has drawn significant attention [1,2]. Robot tasks can be divided into sequencing and continuous tasks [3].…”