Coordinated Reinforcement Learning for Optimizing Mobile Networks

Bouton, Maxime; Farooq, Hasan; Forgeat, Julien; Bothe, Shruti; Shirazipour, Meral; Karlsson, Per

doi:10.48550/arxiv.2109.15175

Cited by 4 publications

(6 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The authors in Ref. [65] treat the coverage and capacity objectives as black-box functions, with no analytical formula and no gradient observations. They identified the set of Pareto optimal solutions through Bayesian optimization (BO) and the deep deterministic policy gradient algorithm (DDPG).…”

Section: Learning-based Methodsmentioning

confidence: 99%

“…In addition to MAB, reinforcement learning is also widely used in online network optimization [64][65][66][67][68] . Dandarov et al [64] proposed an RL approach awarded by the sum data rate normalized to the sector capacity and the number of satisfied users normalized to the potential total number of served users to address CCO problem.…”

Section: Learning-based Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Real‐World Wireless Network Modeling and Optimization: From Model/Data‐Driven Perspective

Yang

Zhang

Ren

et al. 2022

Chinese J of Electronics

View full text Add to dashboard Cite

With the rapid development of the fifthgeneration wireless communication systems, a profound revolution in terms of transmission capacity, energy efficiency, reliability, latency, and connectivity is highly expected to support a new batch of industries and applications. To achieve this goal, wireless networks are becoming extremely dynamic, heterogeneous, and complex. The modeling and optimization for the performance of realworld wireless networks are extremely challenging due to the difficulty to predict the network performance as a function of network parameters, and the prohibitively huge number of parameters to optimize. The conventional network modeling and optimization approaches rely on drive test, trial‐and‐error, and engineering experience, which are labor intensive, error‐prone, and far from optimal. On the other hand, while the research community has spent significant efforts in understanding the fundamental limits of radio channels and developing physical layer techniques to operate close to it, very little is known about the performance limits of wireless networks, where millions of radio channels interact with one another in complex manners. This paper reviews the very recent mathematical and learning based techniques for modeling and optimizing the performance of real‐world wireless networks in five aspects, including channel modeling, user demand and traffic modeling, throughput modeling and prediction, network parameter optimization, and IRS empowered performance optimization, and also presents the corresponding notable performance gains.

show abstract

Section: Learning-based Methodsmentioning

confidence: 99%

Section: Learning-based Methodsmentioning

confidence: 99%

Real‐World Wireless Network Modeling and Optimization: From Model/Data‐Driven Perspective

Yang

Zhang

Ren

et al. 2022

Chinese J of Electronics

View full text Add to dashboard Cite

show abstract

“…The model takes as input a feature vector for each agent to control along with a graph representation of the mobile network. The construction of such graph is described in the previous section and has been also demonstrated in previous work [10], [11]. Each agent is represented by a node in the graph.…”

Section: Graph Q-networkmentioning

confidence: 98%

“…However, they also require an ad-hoc engineering of the reward and can only control one base station at a time [10]. Other algorithms attempting to address the global network optimization problem have been proposed in previous works using coordination graphs [11]. This solution also required a heuristic to handle credit assignment between base stations by splitting individual rewards across neighbors.…”

Section: Related Workmentioning

confidence: 99%

Decomposition methods with deep corrections for reinforcement learning

Bouton

Julian

Nakhaei

et al. 2019

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

Decomposition methods have been proposed to approximate solutions to large sequential decision making problems. In contexts where an agent interacts with multiple entities, utility decomposition can be used to separate the global objective into local tasks considering each individual entity independently. An arbitrator is then responsible for combining the individual utilities and selecting an action in real time to solve the global problem. Although these techniques can perform well empirically, they rely on strong assumptions of independence between the local tasks and sacrifice the optimality of the global solution. This paper proposes an approach that improves upon such approximate solutions by learning a correction term represented by a neural network. We demonstrate this approach on a fisheries management problem where multiple boats must coordinate to maximize their catch over time as well as on a pedestrian avoidance problem for autonomous driving. In each problem, decomposition methods can scale to multiple boats or pedestrians by using strategies involving one entity. We verify empirically that the proposed correction method significantly improves the decomposition method and outperforms a policy trained on the full scale problem without utility decomposition. decomposition methods with deep corrections for reinforcement learning 2 bined in real-time through an arbitrator that maximizes the expected utility [22]. Applications to aircraft collision avoidance with multiple intruders explore summing or using the minimum state-action values [7], [20]. While these approaches tend to perform well empirically, they sacrifice optimality. The solution to each subproblem assumes that its individual policy will be followed regardless of other entities, in contrast to the global policy considering all subproblems [23].Previous approaches to scale decision algorithms using decomposition methods relied on a distributed agent architecture [22]. Each agent is responsible for addressing one of the multiple objectives required to achieve a complex task. At each time step an arbitrator must decide between the different actions recommended by these agents and address possible conflicts. Possible arbitration strategies are command fusion [7], voting [22], lexicographic ordering [33] or utility fusion [22], [23]. It has been shown that utility fusion offers a more principled way of deciding between the individual agents compared to a voting-based approach or command fusion [22]. An alternative approach in leader follower scenarios consists of reducing the problem of controlling the group of agent to controlling a single group leader [15]. Other approaches rely on distributed learning algorithms, such as independent Q-learning, where each agent learns a policy without being aware of the other agents actions [28]. The underlying assumption of independence between agents in these algorithms trades off the benefit of collaboration for an easier learning of the task [8].Utility decomposition methods have been used in many practic...

show abstract

“…Cooperative multi-agent reinforcement learning (MARL) where a team of agents learn coordinated policies optimizing global team rewards has been extensively studied in recent years [25,13], and find potential applications in a wide variety of domains like robot swarm control [15,2], coordinating autonomous drivers [26,41], network routing [38,4], etc. Although cooperative MARL problems can be framed as a centralized single-agent, with the team as that actor with the joint action space, such an approach doesn't scale well.…”

Section: Introductionmentioning

confidence: 99%

Agent-Time Attention for Sparse Rewards Multi-Agent Reinforcement Learning

She¹,

Gupta²,

Kochenderfer³

2022

Preprint

View full text Add to dashboard Cite

Sparse and delayed rewards pose a challenge to single agent reinforcement learning. This challenge is amplified in multi-agent reinforcement learning (MARL) where credit assignment of these rewards needs to happen not only across time, but also across agents. We propose Agent-Time Attention (ATA), a neural network model with auxiliary losses for redistributing sparse and delayed rewards in collaborative MARL. We provide a simple example that demonstrates how providing agents with their own local redistributed rewards and shared global redistributed rewards motivate different policies. We extend several MiniGrid environments, specifically MultiRoom and DoorKey, to the multi-agent sparse delayed rewards setting. We demonstrate that ATA outperforms various baselines on many instances of these environments. Source code of the experiments is available at https://github.com/jshe/agent-time-attention.

show abstract

Coordinated Reinforcement Learning for Optimizing Mobile Networks

Cited by 4 publications

References 17 publications

Real‐World Wireless Network Modeling and Optimization: From Model/Data‐Driven Perspective

Real‐World Wireless Network Modeling and Optimization: From Model/Data‐Driven Perspective

Decomposition methods with deep corrections for reinforcement learning

Agent-Time Attention for Sparse Rewards Multi-Agent Reinforcement Learning

Contact Info

Product

Resources

About