Motion Switching With Sensory and Instruction Signals by Designing Dynamical Systems Using Deep Neural Network

Abstract

To ensure that a robot is able to accomplish an extensive range of tasks, it is necessary to achieve a flexible combination of multiple behaviors. This is because the design of task motions suited to each situation would become increasingly difficult as the number of situations and the types of tasks performed by them increase. To handle the switching and combination of multiple behaviors, we propose a method to design dynamical systems based on point attractors that accept (i) "instruction signals" for instruction-driven switching. We incorporate the (ii) "instruction phase" to form a point attractor and divide the target task into multiple subtasks. By forming an instruction phase that consists of point attractors, the model embeds a subtask in the form of trajectory dynamics that can be manipulated using sensory and instruction signals. Our model comprises two deep neural networks: a convolutional autoencoder and a multiple time-scale recurrent neural network. In this study, we apply the proposed method to manipulate soft materials. To evaluate our model, we design a cloth-folding task that consists of four subtasks and three patterns of instruction signals, which indicate the direction of motion. The results depict that the robot can perform the required task by combining subtasks based on sensory and instruction signals. And, our model determined the relations among these signals using its internal dynamics.

Development of a Basic Educational Kit for Robotic System with Deep Neural Networks

“…End-to-end robot motion learning is currently being studied as a promising approach to perform difficult robot tasks, such as manipulating flexible objects [ 3 , 44 ]. Although we used DNNs only for the recognition part, Refs.…”

Location Instruction-Based Motion Generation for Sequential Robotic Manipulation

“…targets series such as [G 1 , G 2 , G 3 ] are manually input to finish the object stacking task and the manipulation trajectories are obtained as T 1,2,3 = π (O, I(O, G 1,2,3 )). One-hot vector is often used to guide the robot to execute different actions in the same surroundings [31], [32]. Therefore, we choose the method with one-hot vector-based instruction as the baseline and compare it with the proposed framework in the same application scenario.…”

“…For the situation with multiple targets in the perception data, such as sequential manipulation tasks, the robot cannot map the perception information to motion action directly and it may execute different motion actions according to different targets to manipulate. High-level actions or input instructions are needed to eliminate the ambiguity in the mapping from perception to motion action [32]. High-level actions can be considered as the interfaces of the relevant task planning modules and input instructions can be regarded as the interfaces of humans and robots.…”

“…One-hot vectors are often utilized to denote these high-level actions or different tasks [32]. One-hot vectorbased frameworks require the networks of motion generation to have the ability of semantic understanding so that the target objects in the image and the one-hot vectors can be connected.…”

Composable Instructions and Prospection Guided Visuomotor Control for Robotic Manipulation

“…One-hot vector was used to stand for different tasks. It was also used for motion switching in multi-step task such as cloth-folding [45]. Abolghasemi et al improved the robustness of this multi-task approach via task-focused visual attention [15].…”