Q-Learning Algorithms: A Comprehensive Classification and Applications

Abstract

Q-learning is arguably one of the most applied representative reinforcement learning approaches and one of the off-policy strategies. Since the emergence of Q-learning, many studies have described its uses in reinforcement learning and artificial intelligence problems. However, there is an information gap as to how these powerful algorithms can be leveraged and incorporated into general artificial intelligence workflow. Early Q-learning algorithms were unsatisfactory in several aspects and covered a narrow range of applications. It has also been observed that sometimes, this rather powerful algorithm learns unrealistically and overestimates the action values hence abating the overall performance. Recently with the general advances of machine learning, more variants of Q-learning like Deep Q-learning which combines basic Q learning with deep neural networks have been discovered and applied extensively. In this paper, we thoroughly explain how Q-learning evolved by unraveling the mathematical complexities behind it as well its flow from reinforcement learning family of algorithms. Improved variants are fully described, and we categorize Q-learning algorithms into single-agent and multi-agent approaches. Finally, we thoroughly investigate up-to-date research trends and key applications that leverage Q-learning algorithms.

Q-LBR: Q-Learning Based Load Balancing Routing for UAV-Assisted VANET

“…In recent years, artificial intelligence techniques, which include machine learning, have attracted a significant amount of interest from researchers of various fields [ 8 ]. Among such techniques, reinforcement learning (RL) is being investigated in wireless systems because it provides a solution to optimize the system parameters by learning the surrounding area in a dynamic and complicated wireless environment [ 10 , 11 , 12 ]. Q-learning is a representative RL, and studies on using this approach to allocate routing policies in a dynamically changing network environment have been conducted.…”

Online Radio Pattern Optimization Based on Dual Reinforcement-Learning Approach for 5G URLLC Networks

“…The RML [35,36] is a vital branch of the machine learning. It has been widely applied in real-time decision-making problems such as autonomous driving and robot control.…”

“…The proposed solution incorporates a dual RML approach for joint capacity and latency online optimization. We consider the model-free Q-reinforcement-machine-learning (Q-RML) algorithm [35] for its performance merits and low implementation complexity under a moderate state-action space size. As depicted by Fig.…”

A Reinforcement Learning Approach to Dynamic Scheduling in a Product-Mix Flexibility Environment

“…Some studies [11,16,30,35] have indicated that one of the restrictions of dynamic scheduling using machine learning classification approaches is the lack of a revised mechanism in the dynamic scheduling KB. Recent advances in information and communication technology (ICT) using reinforcement learning (RL) [36] has been widely used [37] in robot motion control [38,39], industrial process control [40,41], and production scheduling [15,[42][43][44][45][46], and internet of things (IoT) [47]. ICT learns the optimal policy through interaction with the environment without the model of the environment.…”