Deep Q-Learning Based Optimization of VLC Systems With Dynamic Time-Division Multiplexing

Abstract

The traditional method to solve nondeterministic-polynomial-time (NP)-hard optimization problems is to apply meta-heuristic algorithms. In contrast, Deep Q Learning (DQL) uses memory of experience and deep neural network (DNN) to choose steps and progress towards solving the problem. The dynamic time-division multiple access (DTDMA) scheme is a viable transmission method in visible light communication (VLC) systems. In DTDMA systems, the time-slots of the users are adjusted to maximize the spectral efficiency (SE) of the system. The users in a VLC network have different channel gains because of their physical locations, and the use of variable time-slots can improve the system performance. In this work, we propose a Markov decision process (MDP) model of the DTDMA-based VLC system. The MDP model integrates into deep Q learning (DQL) and provides information to it according to the behavior of the VLC system and the objective to maximize the SE. When we use the proposed MDP model in deep Q learning with experienced replay algorithm, we provide the light emitting diode (LED)-based transmitter an autonomy to solve the problem so it can adjust the time-slots of users using the data collected by device in the past. The proposed model includes definitions of the state, actions, and rewards based on the specific characteristics of the problem. Simulations show that the performance of the proposed DQL method can produce results that are competitive to the well-known metaheuristic algorithms, such as Simulated Annealing and Tabu search algorithms.

Deep Reinforcement Based Power Allocation for the Max-Min Optimization in Non-Orthogonal Multiple Access

“…The above optimization problem is a non-convex optimization problem for N > 2 [14]. A non-convex optimization problem has many local minima and it is hard to determine the globally optimal solution [14], [24], [2]. Different methods such as evolutionary algorithm (EA) [25], [26], [27], [28], monotonic optimization [2], DRL [2], etc., can be applied to solve them.…”

“…A state-representation should capture the attributes of the system that are relevant to the decision-making [32]. Before discussing the state-representation, we would to define two functions f L (∆ i ) and f U (∆ i ), as follows We can determine the data-rates of users (R i ) using (2). Please note that the users are sorted in the ascending order of the square of their channel gains, i.e., |h…”

“…Q-learning is a popular temporal difference (TD)-based model-free method of RL [2]. It uses a table to store the Q-values of all possibles pairs of states and actions.…”

“…In the recent past, the deep reinforcement learning (DRL) approach has successfully been employed to solve optimization problems in many fields of engineering such as non-convex and non-deterministic polynomial-time (NP)hard optimization problems in wireless communications [5], [6], [7], [8], [2]. Non-orthogonal multiple access technique (NOMA) is an innovative multiple-access method proposed for 5G and beyond networks [9], [10], [11], [12].…”

Refined Path Planning for Emergency Rescue Vehicles on Congested Urban Arterial Roads via Reinforcement Learning Approach

“…With the development of artificial intelligence, algorithms like deep Q-network have been widely used for decision making in various practical problems [18][19][20]. e use of reinforcement learning in path planning is increasing and has provided different goal-oriented path planning for various types of vehicles due to its strong performance and high applicability in decision making of path selection.…”