Off-policy Learning for Remote Electrical Tilt Optimization

Abstract

We address the problem of Remote Electrical Tilt (RET) optimization using off-policy Contextual Multi-Armed-Bandit (CMAB) techniques. The goal in RET optimization is to control the orientation of the vertical tilt angle of the antenna to optimize Key Performance Indicators (KPIs) representing the Quality of Service (QoS) perceived by the users in cellular networks. Learning an improved tilt update policy is hard. On the one hand, coming up with a new policy in an online manner in a real network requires exploring tilt updates that have never been used before, and is operationally too risky. On the other hand, devising this policy via simulations suffers from the simulation-toreality gap. In this paper, we circumvent these issues by learning an improved policy in an offline manner using existing data collected on real networks. We formulate the problem of devising such a policy using the off-policy CMAB framework. We propose CMAB learning algorithms to extract optimal tilt update policies from the data. We train and evaluate these policies on realworld 4G Long Term Evolution (LTE) cellular network data. Our policies show consistent improvements over the rule-based logging policy used to collect the data.

Coordinated Reinforcement Learning for Optimizing Mobile Networks

“…When using Reinforcement learning, the problem of coordination still arises. Considering each antenna as independent learning agents have been used in the past to address the problem of optimizing mobile networks [6], [8], [9], [11], [12], hence failing to capture phenomena like interference. Learning algorithms leveraging coordination can use a centralized controller [7], [13] which does not scale to a large number of agents.…”

“…Existing approaches for network optimization rely on hand-engineered strategies which are suboptimal and hard to scale [2]- [4]. Methods relying on mathematical models [5] or reinforcement learning (RL) are also used for network optimization [6]- [11]. They are more robust and principled.…”

Symbolic Reinforcement Learning for Safe RAN Control

“…However, it is known that the large-scale exploration performed by RL algorithms can sometimes take the system to unsafe states [7]. In the problem of RET optimization, RL has been proven to be an effective framework for KPI optimization due to its self-learning capabilities and adaptivity to potential environment changes [16]. For addressing the safety problem (i.e., to guarantee that the desired KPIs remain in specified bounds) authors in [16] have proposed a statistical approach to empirically evaluate the RET optimization in different baseline policies and in different worst-case scenarios.…”

“…However, they assume that the system dynamics abstraction into an MDP is given, which is challenging in network applications that this demonstration refers to. As mentioned previously, authors in [16] address the safe RET optimization problem, but this approach relies on statistical guarantees and it cannot handle general LTL specifications that we treat with this manuscript.…”