A Reinforcement Learning Method of Solving Markov Decision Processes: An Adaptive Exploration Model Based on Temporal Difference Error

Abstract

Traditional backward recursion methods face a fundamental challenge in solving Markov Decision Processes (MDP), where there exists a contradiction between the need for knowledge of optimal expected payoffs and the inability to acquire such knowledge during the decision-making process. To address this challenge and strike a reasonable balance between exploration and exploitation in the decision process, this paper proposes a novel model known as Temporal Error-based Adaptive Exploration (TEAE). Leveraging reinforcement learning techniques, TEAE overcomes the limitations of traditional MDP solving methods. TEAE exhibits dynamic adjustment of exploration probabilities based on the agent’s performance, on the one hand. On the other hand, TEAE approximates the optimal expected payoff function for subprocesses after specific states and times by integrating deep convolutional neural networks to minimize the temporal difference error between the dual networks. Furthermore, the paper extends TEAE to DQN-PER and DDQN-PER methods, resulting in DQN-PER-TEAE and DDQN-PER-TEAE variants, which not only demonstrate the generality and compatibility of the TEAE model with existing reinforcement learning techniques but also validate the practicality and applicability of the proposed approach in a broader MDP reinforcement learning context. To further validate the effectiveness of TEAE, the paper conducts a comprehensive evaluation using multiple metrics, compares its performance with other MDP reinforcement learning methods, and conducts case studies. Ultimately, simulation results and case analyses consistently indicate that TEAE exhibits higher efficiency, highlighting its potential in driving advancements in the field.

Mean Field Multi-Agent Reinforcement Learning Method for Area Traffic Signal Control

“…Transforming the multi-intersection traffic signal control problem within the region into interactions between individual intersections and neighboring intersections enhances the convergence speed of offline learning algorithms [20]. During the experience replay process, it allows for faster learning of the optimal policy from non-current strategy experiences [21] and leads to a more stable All the algorithms designed in this study utilize the same neural network structure. As depicted in Figure 5, it is evident that the performance significantly improved when employing the Mean Field Reinforcement Learning control method.…”

“…Transforming the multi-intersection traffic signal control problem within the region into interactions between individual intersections and neighboring intersections enhances the convergence speed of offline learning algorithms [20]. During the experience replay process, it allows for faster learning of the optimal policy from non-current strategy experiences [21] and leads to a more stable convergence of the optimal policy [22]. MFQ-ATSC starts converging after 100-150 iterations, while MFAC-ATSC still has a few intersections that have not reached convergence even after 400 iterations.…”

Playing Flappy Bird Based on Motion Recognition Using a Transformer Model and LIDAR Sensor

“…In this way, it is possible to look for relationships in historical data from past to future or from future to past and possibly associate them appropriately with the last timestep. The last timestep represents the source of information in the classical Markov decision process (MDP) [ 26 ]. A state vector representing the local memory of the model is fed to the model’s input.…”