Multimodal Reward Shaping for Efficient Exploration in Reinforcement Learning

Abstract

Maintaining long-term exploration ability remains one of the challenges of deep reinforcement learning (DRL). In practice, the reward shaping-based approaches are leveraged to provide intrinsic rewards for the agent to incentivize motivation. However, most existing IRS modules rely on attendant models or additional memory to record and analyze learning procedures, which leads to high computational complexity and low robustness. Moreover, they overemphasize the influence of a single state on exploration, which cannot evaluate the exploration performance from the global perspective. To tackle the problem, state entropy-based methods are proposed to encourage the agent to visit the state space more equitably. However, the estimation error and sample complexity are prohibitive when handling environments with high-dimensional observation. In this paper, we introduce a novel metric entitled Jain's fairness index (JFI) to replace the entropy regularizer, which requires no additional models or memory. In particular, JFI overcomes the vanishing intrinsic rewards problem and can be generalized into arbitrary tasks. Furthermore, we use a variational auto-encoder (VAE) model to capture the life-long novelty of states. Finally, the global JFI score and local state novelty are combined to form a multimodal intrinsic reward, controlling the exploration extent more precisely. Finally, extensive simulation results demonstrate that our multimodal reward shaping (MMRS) method can achieve higher performance in contrast to other benchmark schemes. Our code is available at our GitHub website 1 .1 https://github.com/yuanmingqi/MMRS Preprint. Under review.

Semantic Communications for 6G Future Internet: Fundamentals, Applications, and Challenges

“…Meanwhile, there are also some autonomous systems, such as autonomous driving and drone navigation, that require intelligent agents with computing and learning capabilities to perform real-time decision-making in dynamic environments based on deep reinforcement learning (DRL). However, DRL still faces some challenges in practical applications, such as slow convergence [97], overfitting problems [98], and poor exploration in complex environments [99]. Collaborative DRL (CDRL) is treated as a promising solution to the above issues, wherein the agents can share their experiences and collaboratively learn the optimal policy for their task [69].…”

Semantic-Aware Collaborative Deep Reinforcement Learning Over Wireless Cellular Networks

“…Deep reinforcement learning (DRL) is viewed as a promising approach for training intelligent IoT agents to tackle complex tasks such as autonomous navigation. However, DRL methods can lead to slow convergence [2], overfitting problems [3], or sub-optimal performance due to poor exploration in complex environments [4]. These challenges limit the applications of DRL for real-time autonomous IoT services where convergence time, generalizability of the learning, and performance are all important factors.…”