Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers

Abstract

We propose to address quadrupedal locomotion tasks using Reinforcement Learning (RL) with a Transformer-based model that learns to combine proprioceptive information and high-dimensional depth sensor inputs. While learning-based locomotion has made great advances using RL, most methods still rely on domain randomization for training blind agents that generalize to challenging terrains. Our key insight is that proprioceptive states only offer contact measurements for immediate reaction, whereas an agent equipped with visual sensory observations can learn to proactively maneuver environments with obstacles and uneven terrain by anticipating changes in the environment many steps ahead. In this paper, we introduce Loco-Transformer, an end-to-end RL method for quadrupedal locomotion that leverages a Transformer-based model for fusing proprioceptive states and visual observations. We evaluate our method in challenging simulated environments with different obstacles and uneven terrain. We show that our method obtains significant improvements over policies with only proprioceptive state inputs, and that Transformer-based models further improve generalization across environments. Our project page with videos is at https://RchalYang.github.io/LocoTransformer .

CPG-RL: Learning Central Pattern Generators for Quadruped Locomotion

“…To improve on the reactive controllers trained only with proprioceptive sensing as input, recent works are integrating vision into the deep reinforcement learning framework. This has allowed for obstacle avoidance [23] as well as more dynamic crossing of rough terrain through the use of sampling from height maps [24], [25]. Additional works have shown gap crossing [26], also with full flight phases learned from pixels and leveraging MPC [27].…”

Human-AI Shared Control via Policy Dissection

“…The legged robots locomotion experiments are conducted on the Pybullet-Unitree-A1 [16] robot. We follow the same environment setting used by Locotransformer robot [68], including terrain shape, obstacle distributions, sensors, reward definition, and termination condition. Two agents are trained.…”

“…This forms the "State w/oH " method. We also include "State+Vision" method [68], which trains a visuomotor policy for legged robot directly in the test environment, enabling it to sidestep obstacles. As shown in Table 2, with human involvement, Policy Dissection successfully transforms the "insensible" policy that is not fit to this task and greatly improves the performance, achieving comparable performance to the agent trained directly for this task.…”

“…We use two environments: the one without obstacles as training environment, and the other one with obstacles as test environment. The environment construction, reward definition, tasks are the same as [68]. Its environment has open-source code at: https://github.com/Mehooz/ vision4leg.…”

“…Gym Environments and Pybullet-A1 Legged RobotsAgents of Gym environments (Walker/Ant/Bidepadel Walker) are trained by SAC and share the same hyperpatrameter setting. Quadrupedal robots are trained by PPO, and we follow the same codebase and configuration used in[68].…”