Humanoids Learning to Walk: A Natural CPG-Actor-Critic Architecture

Abstract

The identification of learning mechanisms for locomotion has been the subject of much research for some time but many challenges remain. Dynamic systems theory (DST) offers a novel approach to humanoid learning through environmental interaction. Reinforcement learning (RL) has offered a promising method to adaptively link the dynamic system to the environment it interacts with via a reward-based value system. In this paper, we propose a model that integrates the above perspectives and applies it to the case of a humanoid (NAO) robot learning to walk the ability of which emerges from its value-based interaction with the environment. In the model, a simplified central pattern generator (CPG) architecture inspired by neuroscientific research and DST is integrated with an actor-critic approach to RL (cpg-actor-critic). In the cpg-actor-critic architecture, least-square-temporal-difference based learning converges to the optimal solution quickly by using natural gradient learning and balancing exploration and exploitation. Futhermore, rather than using a traditional (designer-specified) reward it uses a dynamic value function as a stability indicator that adapts to the environment. The results obtained are analyzed using a novel DST-based embodied cognition approach. Learning to walk, from this perspective, is a process of integrating levels of sensorimotor activity and value.

Snake Robot Controlled by Biomimetic CPGs

“…Biomimetic CPGs [12][13][14][15] are implemented in a digital platform: CMOD A7-A35 Xilinx Artix-7 Field Programmable Gate Array (FPGA). The neural network implementation architecture operates on a single computation core.…”

Sigmoid transition approach of the central pattern generator-based controller for the snake-like robot

“…The rhythmic signals are utilized to control and coordinate each joint simultaneously. [7][8][9] The CPG-based controller has already been applied to the control of humanoid robots, 10 quadruped robots, 11 fish robots, 12 salamander robots 13 and robotic manta rays 14 and so on. The CPG-based controller has also been utilized to control the locomotion of snake-like robots.…”

Implementation of biomimetic central pattern generators on field-programmable gate array

“…Neurobotics uses CPGs to move and coordinate the movements of bipedal (Li et al, 2013), multipedal, (Barron-Zambrano and Torres-Huitzil, 2013) or swimming robots (Ijspeert et al, 2007). Neurobotics uses CPGs to move and coordinate the movements of bipedal (Li et al, 2013), multipedal, (Barron-Zambrano and Torres-Huitzil, 2013) or swimming robots (Ijspeert et al, 2007).…”