A Kind of Joint Routing and Resource Allocation Scheme Based on Prioritized Memories-Deep Q Network for Cognitive Radio Ad Hoc Networks

Abstract

Cognitive Radio (CR) is a promising technology to overcome spectrum scarcity, which currently faces lots of unsolved problems. One of the critical challenges for setting up such systems is how to coordinate multiple protocol layers such as routing and spectrum access in a partially observable environment. In this paper, a deep reinforcement learning approach is adopted for solving above problem. Firstly, for the purpose of compressing huge action space in the cross-layer design problem, a novel concept named responsibility rating is introduced to help decide the transmission power of every Secondary User (SU). In order to deal with problem of dimension curse while reducing replay memory, the Prioritized Memories Deep Q-Network (PM-DQN) is proposed. Furthermore, PM-DQN is applied to solve the joint routing and resource allocation problem in cognitive radio ad hoc network for minimizing the transmission delay and power consumption. Simulation results illustrates that our proposed algorithm can reduce the end-to-end delay, packet loss ratio and estimation error while achieving higher energy efficiency compared with traditional algorithm.

An Energy-Efficient Cross-Layer Routing Protocol for Cognitive Radio Networks Using Apprenticeship Deep Reinforcement Learning

“…Otherwise, it is assumed that the current transmit power is being maintained within an appropriate range and should remain unchanged. According to this principle, the concept of responsibility rating was introduced in [20,23]. Nevertheless, as for the responsibility rating, the update framework is defined as follows:…”

“…In this section, the performance of the PM-DQfD-based scheme is assessed. The results of the proposed scheme are compared with traditional schemes such as i) Deep Q-Network (DQN) [31]; ii) Prioritized Memories Deep Q-Network (PM-DQN) [20]; iii) Natural DQfD [29]; iv) Conjecture Based Multi-agent Q-learning Scheme (CBMQ) [32]; and v) Cognitive Radio Q-routing (CRQ-routing) [33] with respect to effectiveness, robustness and learning speed. The simulation framework was built using Python (3.5.1, Google, Mountain View, CA, USA).…”

“…The input layer is not included in the total number of layers in the neural network because it only transmits data into the network and does not participate in the calculation. The feed-forward calculation for one sample contains two matrix operations, which needs n 1 × n 2 and n 2 × n action times of computation [20]. Since the size of the action space n action is a constant, the time complexity for one sample in the feed-forward calculation can be calculated as o(n 1 × n 2 + n 2 × n action ) = o(n 1 × n 2 ).…”

A Cross-Layer Routing Protocol Based on Quasi-Cooperative Multi-Agent Learning for Multi-Hop Cognitive Radio Networks

“…However, the action space will be fairly large if we treat power assignment as actions in the joint optimization problem. Huge action space results in intensive computation complexity and low learning efficiency due to the maximum calculation in Q-value updating [17]. In this case, the concept called responsibility rating was introduced in our previous work [17].…”

“…In our previous work [17], we designed a single-agent based intelligent joint routing and resource assignment scheme for CRN to achieve the maximum cumulative rewards. In this paper, we adopt a quasi-cooperative multi-agent learning scheme for routing and radio resource management, which is more efficient than the single-agent strategy in multi-hop CRN.…”