DeepGait: Planning and Control of Quadrupedal Gaits Using Deep Reinforcement Learning

Abstract

This paper addresses the problem of legged locomotion in non-flat terrain. As legged robots such as quadrupeds are to be deployed in terrains with geometries which are difficult to model and predict, the need arises to equip them with the capability to generalize well to unforeseen situations. In this work, we propose a novel technique for training neuralnetwork policies for terrain-aware locomotion, which combines state-of-the-art methods for model-based motion planning and reinforcement learning. Our approach is centered on formulating Markov decision processes using the evaluation of dynamic feasibility criteria in place of physical simulation. We thus employ policy-gradient methods to independently train policies which respectively plan and execute foothold and base motions in 3D environments using both proprioceptive and exteroceptive measurements. We apply our method within a challenging suite of simulated terrain scenarios which contain features such as narrow bridges, gaps and stepping-stones, and train policies which succeed in locomoting effectively in all cases. * These authors contributed equally.

Cat-Like Jumping and Landing of Legged Robots in Low Gravity Using Deep Reinforcement Learning

“…We decide to tackle these challenges using model-free control via Reinforcement Learning (RL), which has shown impressive results when it comes to the control of complex motions. It has been used to solve a Rubik's cube with a robotic hand [16], learn locomotion on complex terrains, [17], [18], play table tennis [19], teach robots to imitate animals [20] and stand up from arbitrary initial conditions [21]. In addition to its capacity to solve complex tasks, once trained, RL has the advantage of requiring much less computation than optimization methods.…”

Achieving natural behavior in a robot using neurally inspired hierarchical control

“…Every module, with the exception of the burst generators, independently and continuously seeks to match its perception with its goal value through corrective outputs in real time. Unlike designs that only implement feedback control at the level of joint control and some form of feed-forward computation above that to generate behavior (11,14, 22, 25, 28, 32, 43, 45, 50), our design uses closed loop negative feedback control at every level of the hierarchy. This feature replicates the purposive nature of animal behavior, which is often mistakenly assumed to be a feedforward process whereby a stimulus input is transformed by the nervous system and results in motor output (51).…”

“…The ‘nervous system’ of the robot operates through a hierarchical network of simple control system modules. Unlike other robot control architectures that perform model-based control and planning (11, 12, 13, 14, 22, 23, 25, 33, 45, 50), our control architecture generates robust and adaptive goal-directed behavior through a simple feedback process requiring no model of the environment, prediction of future states, or learning. Unlike architectures in which behavior is generated by environmental stimuli or internal system dynamics (1, 2, 3, 4, 18, 24, 28, 29, 32, 36, 35, 37, 43, 46), our architecture generates adaptive behavior by automatically achieving continuously changing internal goals in the control hierarchy.…”

Adaptive Locomotion Control of a Hexapod Robot via Bio-Inspired Learning

“…The two-layer CPG network is chosen as the locomotion generator instead of learning joint position commands directly like most of the other studies (Hwangbo et al, 2019 ; Tsounis et al, 2020 ). There are three reasons for this: (1) the CPG network constrains the basic locomotion of the robot, which reduces the search space and accelerates the learning; (2) compared to 18 joint position or joint torque commands, learning symmetric CPG coupling parameters lowers the dimension of the action space; (3) the CPG network outputs smooth joint position commands, which are easier to be realized in the real robot.…”