Parameterized MDPs and Reinforcement Learning Problems—A Maximum Entropy Principle-Based Framework

Abstract

We present a framework to address a class of sequential decision making problems. Our framework features learning the optimal control policy with robustness to noisy data, determining the unknown state and action parameters, and performing sensitivity analysis with respect to problem parameters. We consider two broad categories of sequential decision making problems modeled as infinite horizon Markov Decision Processes (MDPs) with (and without) an absorbing state. The central idea underlying our framework is to quantify exploration in terms of the Shannon Entropy of the trajectories under the MDP and determine the stochastic policy that maximizes it while guaranteeing a low value of the expected cost along a trajectory. This resulting policy enhances the quality of exploration early on in the learning process, and consequently allows faster convergence rates and robust solutions even in the presence of noisy data as demonstrated in our comparisons to popular algorithms such as Q-learning, Double Q-learning and entropy regularized Soft Q-learning. The framework extends to the class of parameterized MDP and RL problems, where states and actions are parameter dependent, and the objective is to determine the optimal parameters along with the corresponding optimal policy. Here, the associated cost function can possibly be non-convex with multiple poor local minima. Simulation results applied to a 5G small cell network problem demonstrate successful determination of communication routes and the small cell locations. We also obtain sensitivity measures to problem parameters and robustness to noisy environment data.

Time-Varying Parameters in Sequential Decision Making Problems

“…Remark 1. To ensure that the cumulative cost J µ ζη (s) is finite for all s ∈ S and the system reaches the cost-free termination state δ in finite steps, we assume that there exists atleast one proper policy μ(a|s) ∈ {0, 1} ∀a ∈ A, s ∈ S, and for all parameter values in ζ and η, under which there is a non-zero probability to reach the cost-free termination state δ starting from any state s ∈ S (please see [12] for proof ).…”

“…Further, due to the additional state and action parameters it is difficult to model para-SDMs directly by using the existing frameworks [2,3,4,5] that model SDMs. We have addressed the static para-SDMs in [12].…”

“…One of the main contributions of our earlier work [12] on para-SDM is to view them as combinatorial optimization problems. This is owing to the combinatorially large number of possible sequences of states and actions (also, referred to as paths) in the SDM.…”

“…This viewpoint enables the use of Maximum Entropy Principle (MEP) -from statistical physics literature [13] -in addressing para-SDM problems; where in prior literature MEP has proven itself successful in addressing a wide variety of combinatorial optimization problems such as facility location problem [14], protein structure alignment [15], and graph and markov chain aggregation [16]. In brief, the MEP-based framework proposed in [12] simultaneously determines a distribution over the paths, and the parameter values that maximize the associated Shannon entropy [13] of the distribution; while ensuring that the expected cumulative cost of the para-SDM attains a pre-specified value. The framework, then, employs an iterative scheme (an annealing scheme) and improves upon the distribution over the paths (i.e., the decision policy) as well as the parameter values that correspond to decreasing cumulative cost values of the SDM.…”

“…The optimal decision policy of the SDM, which is also time-varying owing to the evolving parameters Υ(t), is evaluated as the fixed point of the recursive Bellman equation satisfied by underlying cumulative cost function. We build upon the Maximum Entropy Principle (MEP) based framework proposed in [12] that addresses the static para-SDM problems. In particular, this MEPbased framework results into a smooth approximation (also referred to as free-energy) of the cumulative cost, which we exploit as a control-Lyapunov function describing the dynamic para-SDM problems.…”