Optimization for Reinforcement Learning: From a single agent to cooperative agents

Lee, Donghwan; He, Niao; Kamalaruban, Parameswaran; Cevher, Volkan

doi:10.1109/msp.2020.2976000

Cited by 85 publications

(46 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MARL can be classified into two cases: MARLs with centralized or decentralized reward. In MARL with centralized rewards, all agents receive a common (central) reward, on the other hand in MARL with decentralized, every agent obtains a distinct reward [26]. However, in multiagent environment, all agents under decentralized way may compete with each other, i.e., agents may act in a selfish behaviour for requiring the highest reward which may effect the global network performance.…”

Section: A Overview Of Multi-agent Deep Reinforcement Learningmentioning

confidence: 99%

Transmit Power Pool Design for Grant-Free NOMA-IoT Networks via Deep Reinforcement Learning

Fayaz

Liu

et al. 2021

IEEE Trans. Wireless Commun.

View full text Add to dashboard Cite

Grant-free non-orthogonal multiple access (GF-NOMA) is a potential multiple access framework for shortpacket internet-of-things (IoT) networks to enhance connectivity. However, the resource allocation problem in GF-NOMA is challenging due to the absence of closed-loop power control. We design a prototype of transmit power pool (PP) to provide open-loop power control. IoT users acquire their transmit power in advance from this prototype PP solely according to their communication distances. Firstly, a multi-agent deep Q-network (DQN) aided GF-NOMA algorithm is proposed to determine the optimal transmit power levels for the prototype PP. More specifically, each IoT user acts as an agent and learns a policy by interacting with the wireless environment that guides them to select optimal actions. Secondly, to prevent the Q-learning model overestimation problem, double DQN (DDQN) based GF-NOMA algorithm is proposed. Numerical results confirm that the DDQN based algorithm finds out the optimal transmit power levels that form the PP. Comparing with the conventional online learning approach, the proposed algorithm with the prototype PP converges faster under changing environments due to limiting the action space based on previous learning. The considered GF-NOMA system outperforms the networks with fixed transmission power, namely all the users have the same transmit power and the traditional GF with orthogonal multiple access techniques, in terms of throughput.

show abstract

Section: A Overview Of Multi-agent Deep Reinforcement Learningmentioning

confidence: 99%

Transmit Power Pool Design for Grant-Free NOMA-IoT Networks via Deep Reinforcement Learning

Fayaz

Liu

et al. 2021

IEEE Trans. Wireless Commun.

View full text Add to dashboard Cite

show abstract

“…The application of MARL in solving problems for vehicular networks is studied in [85]. In [86], an overview of the evolution of cooperative MARL algorithms is presented with an emphasis on distributed optimization.…”

Section: E Multi-agent Drl Algorithmsmentioning

confidence: 99%

Deep Reinforcement Learning for Radio Resource Allocation and Management in Next Generation Heterogeneous Wireless Networks: A Survey

Alwarafy¹,

Abdallah²,

Çiftler³

et al. 2021

Preprint

View full text Add to dashboard Cite

<div>Next generation wireless networks are expected to be extremely complex due to their massive heterogeneity in terms of the types of network architectures they incorporate, the types and numbers of smart IoT devices they serve, and the types of emerging applications they support. In such large-scale and heterogeneous networks (HetNets), radio resource allocation and management (RRAM) becomes one of the major challenges encountered during system design and deployment. In this context, emerging Deep Reinforcement Learning (DRL) techniques are expected to be one of the main enabling technologies to address the RRAM in future wireless HetNets. In this paper, we conduct a systematic in-depth, and comprehensive survey of the applications of DRL techniques in RRAM for next generation wireless networks. Towards this, we first overview the existing traditional RRAM methods and identify their limitations that motivate the use of DRL techniques in RRAM. Then, we provide a comprehensive review of the most widely used DRL algorithms to address RRAM problems, including the value- and policy-based algorithms. The advantages, limitations, and use-cases for each algorithm are provided. We then conduct a comprehensive and in-depth literature review and classify existing related works based on both the radio resources they are addressing and the type of wireless networks they are investigating. To this end, we carefully identify the types of DRL algorithms utilized in each related work, the elements of these algorithms, and the main findings of each related work. Finally, we highlight important open challenges and provide insights into several future research directions in the context of DRL-based RRAM. This survey is intentionally designed to guide and stimulate more research endeavors towards building efficient and fine-grained DRL-based RRAM schemes for future wireless networks.</div>

show abstract

“…Challenges in Multi-agent Reinforcement Learning Moving from a single-agent to a multi-agent environment brings about new complex challenges with respect to learning and evaluating outcomes. This can be attributed to several factors, mainly including the exponential growth of the search space and the non-stationary of the environment [127]. In the following section the additional challenges for MADRL applications will be discussed.…”

Section: ) Multi-agent Reinforcement Learningmentioning

confidence: 99%

Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey

Majid¹

2021

Preprint

View full text Add to dashboard Cite

<div>Deep Reinforcement Learning (DRL) has the potential to surpass human-level control in sequential decision-making problems. Evolution Strategies (ESs) have different characteristics than DRL, yet they are promoted as a scalable alternative. </div><div>To get insights into their strengths and weaknesses, in this paper, we put the two approaches side by side. After presenting the fundamental concepts and algorithms for each of the two approaches, they are compared from the perspectives of scalability, exploration, adaptation to dynamic environments, and multi-agent learning. Then, the paper discusses hybrid algorithms, combining aspects of both DRL and ESs, and how they attempt to capitalize on the benefits of both techniques. Lastly, both approaches are compared based on the set of applications they support, showing their potential for tackling real-world problems.</div><div>This paper aims to present an overview of how DRL and ESs can be used, either independently or in unison, to solve specific learning tasks. It is intended to guide researchers to select which method suits them best and provides a bird's eye view of the overall literature in the field. Further, we also provide application scenarios and open challenges. </div>

show abstract

Optimization for Reinforcement Learning: From a single agent to cooperative agents

Cited by 85 publications

References 28 publications

Transmit Power Pool Design for Grant-Free NOMA-IoT Networks via Deep Reinforcement Learning

Transmit Power Pool Design for Grant-Free NOMA-IoT Networks via Deep Reinforcement Learning

Deep Reinforcement Learning for Radio Resource Allocation and Management in Next Generation Heterogeneous Wireless Networks: A Survey

Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey

Contact Info

Product

Resources

About