In-air Knotting of Rope using Dual-Arm Robot based on Deep Learning

Abstract

In this study, we report the successful execution of in-air knotting of rope using a dual-arm two-finger robot based on deep learning. Owing to its flexibility, the state of the rope was in constant flux during the operation of the robot. This required the robot control system to dynamically correspond to the state of the object at all times. However, a manual description of appropriate robot motions corresponding to all object states is difficult to be prepared in advance. To resolve this issue, we constructed a model that instructed the robot to perform bowknots and overhand knots based on two deep neural networks trained using the data gathered from its sensorimotor, including visual and proximity sensors. The resultant model was verified to be capable of predicting the appropriate robot motions based on the sensory information available online. In addition, we designed certain task motions based on the Ian knot method using the dual-arm two-fingers robot. The designed knotting motions do not require a dedicated workbench or robot hand, thereby enhancing the versatility of the proposed method. Finally, experiments were performed to estimate the knotting performance of the real robot while executing overhand knots and bowknots on rope and its success rate. The experimental results established the effectiveness and high performance of the proposed method.

Rapid prototyping for series of tasks in atypical environment: robotic system with reliable program-based and flexible learning-based approaches

“…Specifically, it consists of three steps: (1) collect sensory-motor information (e.g., camera image, joint angle, and torque) with the robot as learning data when a human teleoperates the robot or performs direct teaching, (2) input the sensor information x t at time t into the model, output the sensor information ŷt+1 at the next time t + 1 , and update the weights of the model to minimize the error between the predicted value ŷt+1 and the true value x t+1 , and (3) at execution time, the robot is made to generate sequential motions by inputting the robot's sensor information x t and inputting the predicted value (motion command value) to the robot for the next time. This method can be used to perform various tasks, such as flexible object handling, which is difficult to do with the conventional method [31,32].…”

Efficient multitask learning with an embodied predictive model for door opening and entry with whole-body control

“…The retractor function of the trained RNN model performs perceptual inference (the fusion of the predicted sensory and current sensory inputs) and active inference (behavioral adjustment), resulting in an attractor transition that reduces its prediction error. As a result, multiple tasks-such as handling flexible objects (24,25), liquids, and powder-are realized in the real world with a multi-degree-of-freedom robot. We also realized flexible contactbased object manipulation that was difficult to control with vision alone (26).…”