Deep reinforcement learning for real-world quadrupedal locomotion: a comprehensive review

Zhang, Hongyin; Li, He; Wang, Donglin

doi:10.20517/ir.2022.20

Cited by 5 publications

(4 citation statements)

References 74 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, such an approach would require re-tuning the hyperparameters from scratch each time we face a new task. It is well-known that tuning hyperparameters for quadruped robot control problems is an extremely complex and tedious process [10]. This would cause our method to lose its plug-and-play capability.…”

Section: Loss Of Plasticity and Re-initializationmentioning

confidence: 99%

Continual Reinforcement Learning for Quadruped Robot Locomotion

Gai,

Lyu,

Zhang

et al. 2024

Entropy

View full text Add to dashboard Cite

The ability to learn continuously is crucial for a robot to achieve a high level of intelligence and autonomy. In this paper, we consider continual reinforcement learning (RL) for quadruped robots, which includes the ability to continuously learn sub-sequential tasks (plasticity) and maintain performance on previous tasks (stability). The policy obtained by the proposed method enables robots to learn multiple tasks sequentially, while overcoming both catastrophic forgetting and loss of plasticity. At the same time, it achieves the above goals with as little modification to the original RL learning process as possible. The proposed method uses the Piggyback algorithm to select protected parameters for each task, and reinitializes the unused parameters to increase plasticity. Meanwhile, we encourage the policy network exploring by encouraging the entropy of the soft network of the policy network. Our experiments show that traditional continual learning algorithms cannot perform well on robot locomotion problems, and our algorithm is more stable and less disruptive to the RL training progress. Several robot locomotion experiments validate the effectiveness of our method.

show abstract

Section: Loss Of Plasticity and Re-initializationmentioning

confidence: 99%

Continual Reinforcement Learning for Quadruped Robot Locomotion

Gai,

Lyu,

Zhang

et al. 2024

Entropy

View full text Add to dashboard Cite

show abstract

“…[18] PG-based algorithms maximize the action policy' s expected return by updating the action policy directl [19] . The PPO algorithm' s main objective is [18] 𝐿 (𝜃) = E [𝐿 𝑎𝑐𝑡𝑜𝑟 (𝜃) − 𝑐 1 𝐿 𝑐𝑟𝑖𝑡𝑖𝑐 (𝜃) + 𝑐 2 𝑆 𝜃 (𝑜 𝑡 )] , (11) where 𝐿 𝑎𝑐𝑡𝑜𝑟 and 𝐿 𝑐𝑟𝑖𝑡𝑖𝑐 are the loss function for actor network and critic network, 𝜃 is the parameters of networks, 𝑐 1 and 𝑐 2 are coefficients, 𝑆 𝜃 is the entropy bonus which is used to ensure sufficient exploration, and 𝑜 𝑡 is the observation of the actor network [18] . By considering these terms, the loss calculated by Equation ( 11) is related to the parameters of actor and critic networks.…”

Section: The Ppo Algorithmmentioning

confidence: 99%

“…Many types of DRL algorithms have been proposed, like deep Q network (DQN), deep deterministic policy gradient (DDPG), proximal policy optimization (PPO), etc. DRL has been applied in UAV path control [10] , quadrupedal locomotion control [11] , autonomous platoon control [12] , etc. At the same time, DRL has been used to improve the operational efficiency of air combat [9,13,14] .…”

Section: Introductionmentioning

confidence: 99%

UAV maneuver decision-making via deep reinforcement learning for short-range air combat

Zhang¹,

Duan²

2023

Intell Robot

View full text Add to dashboard Cite

The unmanned aerial vehicle (UAV) has been applied in unmanned air combat because of its flexibility and practicality. The short-range air combat situation is rapidly changing, and the UAV has to make the autonomous maneuver decision as quickly as possible. In this paper, a type of short-range air combat maneuver decision method based on deep reinforcement learning is proposed. Firstly, the combat environment, including UAV motion model and the position and velocity relationships, is described. On this basic, the combat process is established. Secondly, some improved points based on proximal policy optimization (PPO) are proposed to enhance the maneuver decision-making ability. The gate recurrent unit (GRU) can help PPO make decisions with continuous timestep data. The actor network's input is the observation of UAV, however, the input of the critic network, named state, includes the blood values which cannot be observed directly. In addition, the action space with 15 basic actions and well-designed reward function are proposed to combine the air combat environment and PPO. In particular, the reward function is divided into dense reward, event reward and end-game reward to ensure the training feasibility. The training process is composed of three phases to shorten the training time. Finally, the designed maneuver decision method is verified through the ablation study and confrontment tests. The results show that the UAV with the proposed maneuver decision method can obtain an effective action policy to make a more flexible decision in air combat.

show abstract

“…Furthermore, they designed a defogging algorithm and an edge-fitting term to improve the segmentation process, ultimately achieving better results. In 2012, with the ascent of convolutional networks, artificial intelligence [10] has experienced rapid growth and advancement, and it has gradually found its way into various aspects of human life. Long J.…”

Section: Related Researchmentioning

confidence: 99%

U-Net-Embedded Gabor Kernel and Coaxial Correction Methods to Dorsal Hand Vein Image Projection System

Chen,

Lv,

Cai

et al. 2023

Applied Sciences

View full text Add to dashboard Cite

Vein segmentation and projection correction constitute the core algorithms of an auxiliary venipuncture device, responding to accurate venous positioning to assist puncture and reduce the number of punctures and pain of patients. This paper proposes an improved U-Net for segmenting veins and a coaxial correction for image alignment in the self-built vein projection system. The proposed U-Net is embedded by Gabor convolution kernels in the shallow layers to enhance segmentation accuracy. Additionally, to mitigate the semantic information loss caused by channel reduction, the network model is lightweighted by means of replacing conventional convolutions with inverted residual blocks. During the visualization process, a method that combines coaxial correction and a homography matrix is proposed to address the non-planarity of the dorsal hand in this paper. First, we used a hot mirror to adjust the light paths of both the projector and the camera to be coaxial, and then aligned the projected image with the dorsal hand using a homography matrix. Using this approach, the device requires only a single calibration before use. With the implementation of the improved segmentation method, an accuracy rate of 95.12% is achieved by the dataset. The intersection-over-union ratio between the segmented and original images is reached at 90.07%. The entire segmentation process is completed in 0.09 s, and the largest distance error of vein projection onto the dorsal hand is 0.53 mm. The experiments show that the device has reached practical accuracy and has values of research and application.

show abstract

Deep reinforcement learning for real-world quadrupedal locomotion: a comprehensive review

Cited by 5 publications

References 74 publications

Continual Reinforcement Learning for Quadruped Robot Locomotion

Continual Reinforcement Learning for Quadruped Robot Locomotion

UAV maneuver decision-making via deep reinforcement learning for short-range air combat

U-Net-Embedded Gabor Kernel and Coaxial Correction Methods to Dorsal Hand Vein Image Projection System

Contact Info

Product

Resources

About