Online Markov decision processes with Kullback-Leibler control cost

Guan, Peng; Raginsky, Maxim; Willett, Rebecca

doi:10.1109/acc.2012.6314926

Cited by 6 publications

(3 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We are aware of only a few cases where (relative) entropy regularization has been combined with Markov decision processes and related models. [21] consider a generalization of the Markov decision process where, instead of impacting the process through some actions, the agent can directly manipulate the transition matrix of the system state. However, such manipulation would incur some cost which is proportional to the relative entropy between the transition probability after manipulation, and the transition matrix of a 'passive' process which models the 'natural' system evolution.…”

Section: Related Workmentioning

confidence: 99%

Modeling and optimization of Human-Machine Interaction Processes via the Maximum Entropy Principle

Zheng

Veciana

2019

2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton)

View full text Add to dashboard Cite

We propose a data-driven framework to enable the modeling and optimization of human-machine interaction processes, e.g., systems aimed at assisting humans in decision-making or learning, work-load allocation, and interactive advertising. This is a challenging problem for several reasons. First, humans' behavior is hard to model or infer, as it may reflect biases, long term memory, and sensitivity to sequencing, i.e., transience and exponential complexity in the length of the interaction. Second, due to the interactive nature of such processes, the machine policy used to engage with human may bias possible data-driven inferences. Finally, in choosing machine policies that optimize interaction rewards, one must, on the one hand, avoid being overly sensitive to error/variability in the estimated human model, and on the other, being overly deterministic/predictable which may result in poor human 'engagement' in the interaction. To meet these challenges, we propose a robust approach, based on the maximum entropy principle, which iteratively estimates human behavior and optimizes the machine policy-Alternating Entropy-Reward Ascent (AREA) algorithm. We characterize AREA, in terms of its space and time complexity and convergence. We also provide an initial validation based on synthetic data generated by an established noisy nonlinear model for human decision-making.

show abstract

Section: Related Workmentioning

confidence: 99%

Modeling and optimization of Human-Machine Interaction Processes via the Maximum Entropy Principle

Zheng

Veciana

2019

2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton)

View full text Add to dashboard Cite

show abstract

“…Given a randomized stationary policy π with stationary state distribution d π , the MDP is a Markov chain with transition matrix P π given by (3). Thus, it must satisfy the following balance equation:…”

Section: Ocmdp Algorithm 21 Preliminariesmentioning

confidence: 99%

Online Learning in Weakly Coupled Markov Decision Processes

WeiXiaohan¹,

YuHao²,

NeelyMichael³

2018

SIGMETRICS Perform. Eval. Rev.

View full text Add to dashboard Cite

We consider multiple parallel Markov decision processes (MDPs) coupled by global constraints, where the time varying objective and constraint functions can only be observed after the decision is made. Special attention is given to how well the decision maker can perform in T slots, starting from any state, compared to the best feasible randomized stationary policy in hindsight. We develop a new distributed online algorithm where each MDP makes its own decision each slot after observing a multiplier computed from past information. While the scenario is significantly more challenging than the classical online learning context, the algorithm is shown to have a tight O (√ T) regret and constraint violations simultaneously. To obtain such a bound, we combine several new ingredients including ergodicity and mixing time bound in weakly coupled MDPs, a new regret analysis for online constrained optimization, a drift analysis for queue processes, and a perturbation analysis based on Farkas' Lemma.

show abstract

“…These works, by leveraging a similar framework, formalize the control problem as the (unconstrained) problem of minimizing a cost that captures the discrepancy between an ideal probability density function and the actual probability density function of the system under control. An online version of these algorithms has been proposed in Guan et al (2014): in such a work, by leveraging an average cost formulation, the probability mass function for the state transitions is found. Finally, we also recall here Russo (2021); Garrabe and Russo (2022), where policies are obtained from the minimization of similar costs by leveraging multiple, specialized, datasets.…”

Section: Introductionmentioning

confidence: 99%

Discrete fully probabilistic design: towards a control pipeline for the synthesis of policies from examples

Ferrentino¹,

Chiacchio²,

Russo³

2021

Preprint

View full text Add to dashboard Cite

We present a discretized design that expounds an algorithm recently introduced in Gagliardi and Russo (2021) to synthesize control policies from examples for constrained, possibly stochastic and nonlinear, systems. The constraints do not need to be fulfilled in the possibly noisy example data, which in turn might be collected from a system that is different from the one under control. For this discretized design, we discuss a number of properties and give a design pipeline. The design, which we term as discrete fully probabilistic design, is benchmarked numerically on an example that involves controlling an inverted pendulum with actuation constraints starting from data collected from a physically different pendulum that does not satisfy the system-specific actuation constraints.

show abstract

Online Markov decision processes with Kullback-Leibler control cost

Cited by 6 publications

References 36 publications

Modeling and optimization of Human-Machine Interaction Processes via the Maximum Entropy Principle

Modeling and optimization of Human-Machine Interaction Processes via the Maximum Entropy Principle

Online Learning in Weakly Coupled Markov Decision Processes

Discrete fully probabilistic design: towards a control pipeline for the synthesis of policies from examples

Contact Info

Product

Resources

About