A Markov Decision Process based flow assignment framework for heterogeneous network access

Abstract

We consider a scenario where devices with multiple networking capabilities access networks with heterogeneous characteristics. In such a setting, we address the problem of efficient utilization of multiple access networks by devices via optimal assignment of traffic flows with given utilities to different networks. We develop and analyze a device middleware functionality that monitors network characteristics and employs a Markov Decision Process (MDP) based control scheme that in conjunction with stochastic characterization of the available bit rate and delay of the networks generates an optimal policy for allocation of flows to different networks. The optimal policy maximizes, under available bit rate and delay constraints on the access networks, a discounted reward which is a function of the flow utilities. The flow assignment policy is periodically updated and is consulted by the flows to dynamically perform network selection during their lifetimes. We perform measurement tests to collect traces of available bit rate and delay characteristics on Ethernet and WLAN networks on a work day in a corporate work environment. We implement our flow assignment framework in ns-2 and simulate the system performance for a set of elastic video-like flows using the collected traces. We demonstrate that the MDP based flow assignment policy leads to significant enhancement in the QoS provisioning (higher rate allocation, lower packet delays and packet loss rates) for the flows and better access network utilization, as compared to policies that allocate flows to different networks using greedy approaches or heuristics like average available bit rate on the networks.

Optimizing network information for radio access technology selection

“…Otherwise, the average reward obviously decreases with increasing blocking costs. We further note that the optimal policy always outperforms the staircase one with S 1 = 0.35 and S 2 = 0.85, denoted as Staircase policy (2). However, when S 1 and S 2 are carefully set to 0.3 and 0.95, the Staircase policy provides an average reward that is closer to the optimal one.…”

Scalable Monitoring Heuristics for Improving Network Latency

“…Similarly, an approach based on Markovian modeling of network path performance and Markov decision processes has been used to determine optimal flow assignment policies for devices with multiple network access [21]. But this work does not consider the problem of parsimonious measurement.…”

A Hybrid Approach for Radio Access Technology Selection in Heterogeneous Wireless Networks

“…In [16], based on the CEA (Constrained Equal Awards) bankruptcy rule, selection decisions try to equally satisfy mobile users: they are assigned the same amount of resources, without exceeding their individual demands. In [20,6,39,25,7,34,42,44,43], a Semi-Markov Decision Process (SMDP) is employed to model the RAT selection decision-making. A set of states, actions, rewards, and transition probabilities are defined.…”