Decentralized Deep Reinforcement Learning for Delay-Power Tradeoff in Vehicular Communications

Abstract

This paper targets at the problem of radio resource management for expected long-term delay-power tradeoff in vehicular communications. At each decision epoch, the road side unit observes the global network state, allocates channels and schedules data packets for all vehicle user equipment-pairs (VUEpairs). The decision-making procedure is modelled as a discretetime Markov decision process (MDP). The technical challenges in solving an optimal control policy originate from highly spatial mobility of vehicles and temporal variations in data traffic. To simplify the decision-making process, we first decompose the MDP into a series of per-VUE-pair MDPs. We then propose an online long short-term memory based deep reinforcement learning algorithm to break the curse of high dimensionality in state space faced by each per-VUE-pair MDP. With the proposed algorithm, the optimal channel allocation and packet scheduling decision at each epoch can be made in a decentralized way in accordance with the partial observations of the global network state at the VUE-pairs. Numerical simulations validate the theoretical analysis and show the effectiveness of the proposed online learning algorithm.

Age of Information Aware Radio Resource Management in Vehicular Networks: A Proactive Deep Reinforcement Learning Perspective

“…It should be noted that optimizing AoI is totally different from queuing delay minimization as in our prior work [26]. The queuing delay can unnecessarily increase the age of an information update [27].…”

“…The time horizon is discretized into scheduling slots, with each being of equal time duration τ and indexed by a positive integer j ∈ N + . For each VUE-pair, the vTx always follows at a fixed distance of ℓ to the vRx, which moves according to a Manhattan mobility model [15], [26]. We let y j k,(vT x) = (y j,1 k,(vT x) , y j,2 k,(vT x) ) and y j k,(vRx) = (y j,1 k,(vRx) , y j,2 k,(vRx) ) denote, respectively, the Euclidean coordinates of geographical More specifically, the channel state H j k = ψ · h(y j k,(vT x) , y j k,(vRx) ) ∈ H over the frequency bands experienced by a VUE-pair k ∈ K during each slot j includes a fast fading component 1 ψ and a path loss h(y j k,(vT x) , y j k,(vRx) ) that applies…”