Improved Dyna-Q: A Reinforcement Learning Method Focused via Heuristic Graph for AGV Path Planning in Dynamic Environments

Liu, Yiyang; Yan, Shuaihua; Zhao, Yang; Song, Chunhe; Li, Fēi

doi:10.3390/drones6110365

Cited by 8 publications

(4 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While Dyna-Q is often more efficient than Q-Learning when it comes to trajectory planning problems, it is also recognized for its low search efficiency, slow convergence speed, and, in some cases, inability to converge in complex dynamic environments. These issues arise due to the sparse reward function and the vast search space involved [46]. Although the Taxi task was not considered a priori as a complex scenario, it is true that the positive reward is quite sparse, since it is only obtained when the agent succeeds in releasing the passenger at the destination.…”

Section: Policy Learningmentioning

confidence: 99%

CARL: A Synergistic Framework for Causal Reinforcement Learning

Méndez-Molina,

Morales,

Sucar

2023

IEEE Access

View full text Add to dashboard Cite

Causal Reinforcement Learning (CRL) is an emerging field where two essential areas for the development of artificial intelligence are integrated. Existing works in the area have shown how causality can contribute to mitigate some of the limitations of reinforcement learning (RL), ranging from data-inefficiency, lack of interpretability, and long learning times, among others. However, how to use reinforcement learning to support causal discovery (CD) has so far been less explored. In this article, we introduce CARL, a Causality-Aware Reinforcement Learning framework for simultaneously learning and using causal models to speedup the police learning in online Markov decision process (MDP) settings. In a synergistic way, our method alternates between: (i) (RL for CD), where it promotes the selection of actions to obtain better causal models in fewer episodes than traditional methods of obtaining data in RL, (ii) (CD), where an score-based algorithm is used to learn causal models and (iii) (RL using CD), where the learned models are used to select actions that speed up the learning of the optimal policy by reducing the number of interactions with the environment. Experiments in simulated environments show that our method achieves better results in policy learning than traditional model-free and model-based algorithms while it is also able to learn the underlying causal models. We also show how the learned causal models can be directly transferred to a similar task of greater complexity reducing significantly the number of episodes to learn an optimal policy. Finally, the method's scalability to high-dimensional states, where the action-value function needs to be represented with deep neural networks, was verified.

show abstract

Section: Policy Learningmentioning

confidence: 99%

CARL: A Synergistic Framework for Causal Reinforcement Learning

Méndez-Molina,

Morales,

Sucar

2023

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Experimental validation indicates the effectiveness of this method in reducing the waiting time for CNC machines and enhancing overall production efficiency. Liu et al [24] applied an improved reinforcement learning method to AGV path planning. They designed a new dynamic reward function and action selection method, ultimately demonstrating through experimental cases that the method can effectively assist AGVs in obtaining better paths.…”

Section: Introductionmentioning

confidence: 99%

Fusion Q-Learning Algorithm for Open Shop Scheduling Problem with AGVs

Wen,

Zhang,

et al. 2024

Mathematics

View full text Add to dashboard Cite

In accordance with the actual production circumstances of enterprises, a scheduling problem model is designed for open-shop environments, considering AGV transport time. A Q-learning-based method is proposed for the resolution of such problems. Based on the characteristics of the problem, a hybrid encoding approach combining process encoding and AGV encoding is applied. Three pairs of actions are constituted to form the action space. Decay factors and a greedy strategy are utilized to perturb the decision-making of the intelligent agent, preventing it from falling into local optima while simultaneously facilitating extensive exploration of the solution space. Finally, the proposed method proved to be effective in solving the open-shop scheduling problem considering AGV transport time through multiple comparative experiments.

show abstract

“…AGVs have advantages such as their high work efficiency, low cost, and high safety. The research on AGVs includes navigation algorithms [9,10], path planning algorithms [11,12], path tracking algorithms [13,14], and vehicle scheduling algorithms [15,16]. Of these, path planning determines the route by which the AGV will move, and plays a fundamental role in determining the operating costs.…”

Section: Introductionmentioning

confidence: 99%

A Novel AGV Path Planning Approach for Narrow Channels Based on the Bi-RRT Algorithm with a Failure Rate Threshold

Wu,

Zhang,

Chi

et al. 2023

Sensors

View full text Add to dashboard Cite

The efficiency of the rapidly exploring random tree (RRT) falls short when efficiently guiding targets through constricted-passage environments, presenting issues such as sluggish convergence speed and elevated path costs. To overcome these algorithmic limitations, we propose a narrow-channel path-finding algorithm (named NCB-RRT) based on Bi-RRT with the addition of our proposed research failure rate threshold (RFRT) concept. Firstly, a three-stage search strategy is employed to generate sampling points guided by real-time sampling failure rates. By means of the balance strategy, two randomly growing trees are established to perform searching, which improves the success rate of the algorithm in narrow channel environments, accelerating the convergence speed and reducing the number of iterations required. Secondly, the parent node re-selection and path pruning strategy are integrated. This shortens the path length and greatly reduces the number of redundant nodes and inflection points. Finally, the path is optimized by utilizing segmented quadratic Bezier curves to achieve a smooth trajectory. This research shows that the NCB-RRT algorithm is better able to adapt to the complex narrow channel environment, and the performance is also greatly improved in terms of the path length and the number of inflection points. Compared with the RRT, RRT* and Bi-RRT algorithms, the success rate is increased by 2400%, 1900% and 11.11%, respectively.

show abstract

Improved Dyna-Q: A Reinforcement Learning Method Focused via Heuristic Graph for AGV Path Planning in Dynamic Environments

Cited by 8 publications

References 30 publications

CARL: A Synergistic Framework for Causal Reinforcement Learning

CARL: A Synergistic Framework for Causal Reinforcement Learning

Fusion Q-Learning Algorithm for Open Shop Scheduling Problem with AGVs

A Novel AGV Path Planning Approach for Narrow Channels Based on the Bi-RRT Algorithm with a Failure Rate Threshold

Contact Info

Product

Resources

About