Entropy Maximization for Constrained Markov Decision Processes

Savas, Yagiz; Ornik, Melkior; Cubuktepe, Murat; Topcu, Ufuk

doi:10.1109/allerton.2018.8636066

Cited by 13 publications

(16 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Well posedness: For the class of proper policy µ ∈ Γ the maximum entropy H µ (s) ∀ s ∈ S is finite as shown in [40], [41]. In short, the existence of a cost-free termination state δ and a non-zero probability to reach it from any state s ∈ S ensures that the maximum entropy is finite.…”

Section: A Problem Formulationmentioning

confidence: 99%

See 1 more Smart Citation

Parameterized MDPs and Reinforcement Learning Problems—A Maximum Entropy Principle-Based Framework

Srivastava

Salapaka

2022

IEEE Trans. Cybern.

View full text Add to dashboard Cite

We present a framework to address a class of sequential decision making problems. Our framework features learning the optimal control policy with robustness to noisy data, determining the unknown state and action parameters, and performing sensitivity analysis with respect to problem parameters. We consider two broad categories of sequential decision making problems modeled as infinite horizon Markov Decision Processes (MDPs) with (and without) an absorbing state. The central idea underlying our framework is to quantify exploration in terms of the Shannon Entropy of the trajectories under the MDP and determine the stochastic policy that maximizes it while guaranteeing a low value of the expected cost along a trajectory. This resulting policy enhances the quality of exploration early on in the learning process, and consequently allows faster convergence rates and robust solutions even in the presence of noisy data as demonstrated in our comparisons to popular algorithms such as Q-learning, Double Q-learning and entropy regularized Soft Q-learning. The framework extends to the class of parameterized MDP and RL problems, where states and actions are parameter dependent, and the objective is to determine the optimal parameters along with the corresponding optimal policy. Here, the associated cost function can possibly be non-convex with multiple poor local minima. Simulation results applied to a 5G small cell network problem demonstrate successful determination of communication routes and the small cell locations. We also obtain sensitivity measures to problem parameters and robustness to noisy environment data.

show abstract

Section: A Problem Formulationmentioning

confidence: 99%

“…In short, the existence of a cost-free termination state δ and a non-zero probability to reach it from any state s ∈ S ensures that the maximum entropy is finite. Please refer to the Theorem 1 in [40] or Proposition 2 in [41] for further details. Remark 1.…”

Section: A Problem Formulationmentioning

confidence: 99%

Parameterized MDPs and Reinforcement Learning Problems—A Maximum Entropy Principle-Based Framework

Srivastava

Salapaka

2022

IEEE Trans. Cybern.

View full text Add to dashboard Cite

show abstract

“…However, as explained in Section IV-C and shown in the following examples, a policy that induces a stochastic process with an arbitrarily large entropy can easily be obtained by introducing constraints on the expected residence time in certain states. Additional motion planning examples are provided in [17].…”

Section: Examplesmentioning

confidence: 99%

“…A preliminary version[17] of this paper relied on Proposition 36 from[3]. This proposition is not valid in general.…”

mentioning

confidence: 99%

Entropy Maximization for Markov Decision Processes Under Temporal Logic Constraints

Savas

Ornik

Cubuktepe

et al. 2020

IEEE Trans. Automat. Contr.

Self Cite

View full text Add to dashboard Cite

We study the problem of synthesizing a policy that maximizes the entropy of a Markov decision process (MDP) subject to a temporal logic constraint. Such a policy minimizes the predictability of the paths it generates, or dually, maximizes the exploration of different paths in an MDP while ensuring the satisfaction of a temporal logic specification. We first show that the maximum entropy of an MDP can be finite, infinite or unbounded. We provide necessary and sufficient conditions under which the maximum entropy of an MDP is finite, infinite or unbounded. We then present an algorithm which is based on a convex optimization problem to synthesize a policy that maximizes the entropy of an MDP. We also show that maximizing the entropy of an MDP is equivalent to maximizing the entropy of the paths that reach a certain set of states in the MDP. Finally, we extend the algorithm to an MDP subject to a temporal logic specification. In numerical examples, we demonstrate the proposed method on different motion planning scenarios and illustrate the relation between the restrictions imposed on the paths by a specification, the maximum entropy, and the predictability of paths.

show abstract

“…The entropy maximization for MDPs is discussed in [17], which can be considered as a special case of the synthesis of the optimal deceptive policy where the reference policy follows every possible path with equal probability. For the synthesis of optimal deceptive policies, we use a method similar to [17] in that we represent a path as a collection of transitions between the states. We explore the synthesis of optimal reference policies, which, to the best of our knowledge, has not been discussed before.…”

Section: Introductionmentioning

confidence: 99%

Optimal Deceptive and Reference Policies for Supervisory Control

Karabag

Ornik

Topcu

2019

2019 IEEE 58th Conference on Decision and Control (CDC)

Self Cite

View full text Add to dashboard Cite

The use of deceptive strategies is important for an agent that attempts not to reveal his intentions in an adversarial environment. We consider a setting in which a supervisor provides a reference policy and expects an agent to follow the reference policy and perform a task. The agent may instead follow a different, deceptive policy to achieve a different task. We model the environment and the behavior of the agent with a Markov decision process, represent the tasks of the agent and the supervisor with linear temporal logic formulae, and study the synthesis of optimal deceptive policies for such agents. We also study the synthesis of optimal reference policies that prevents deceptive strategies of the agent and achieves the supervisor's task with high probability.We show that the synthesis of deceptive policies has a convex optimization problem formulation, while the synthesis of reference policies requires solving a nonconvex optimization problem.

show abstract

Entropy Maximization for Constrained Markov Decision Processes

Cited by 13 publications

References 9 publications

Parameterized MDPs and Reinforcement Learning Problems—A Maximum Entropy Principle-Based Framework

Parameterized MDPs and Reinforcement Learning Problems—A Maximum Entropy Principle-Based Framework

Entropy Maximization for Markov Decision Processes Under Temporal Logic Constraints

Optimal Deceptive and Reference Policies for Supervisory Control

Contact Info

Product

Resources

About