Markov Decision Processes with Multiple Long-run Average Objectives

Abstract

Abstract. We study Markov decision processes (MDPs) with multiple limit-average (or mean-payoff) functions. We consider two different objectives, namely, expectation and satisfaction objectives. Given an MDP with k k k limit-average functions, in the expectation objective the goal is to maximize the expected limit-average value, and in the satisfaction objective the goal is to maximize the probability of runs such that the limit-average value stays above a given vector. We show that under the expectation objective, in contrast to the case of one limit-average function, both randomization and memory are necessary for strategies even for ε ε ε-approximation, and that finite-memory randomized strategies are sufficient for achieving Pareto optimal values. Under the satisfaction objective, in contrast to the case of one limit-average function, infinite memory is necessary for strategies achieving a specific value (i.e. randomized finite-memory strategies are not sufficient), whereas memoryless randomized strategies are sufficient for ε ε ε-approximation, for all ε > 0 ε > 0 ε > 0. We further prove that the decision problems for both expectation and satisfaction objectives can be solved in polynomial time and the trade-off curve (Pareto curve) can be ε ε ε-approximated in time polynomial in the size of the MDP and, and exponential in the number of limitaverage functions, for all ε > 0 ε > 0 ε > 0. Our analysis also reveals flaws in previous work for MDPs with multiple mean-payoff functions under the expectation objective, corrects the flaws, and allows us to obtain improved results. ACM CCS: [Mathematics

Percentile Queries in Multi-dimensional Markov Decision Processes

“…The need for infinite memory was proved in [6,Section 5] for the problem of ensuring thresholds Fig. 4, v 1 = v 2 = 0.5 and α = 1 can be ensured by an infinite-memory strategy and that finite-memory strategies can only achieve these thresholds with probability 0.…”

“…The linear program follows the ideas of [18,6]. Note that the first two lines of (L) corresponds to the multiple reachability LP of [18] for absorbing target states.…”

“…First, there is a series of works that investigate MDPs with multi-dimensional weights [14,6] rather than single-dimensional as it is traditionally the case. Multi-dimensional MDPs are useful to analyze systems with multiple objectives that are potentially conflicting and make necessary the analysis of trade-offs.…”

Trading Performance for Stability in Markov Decision Processes

“…For example, although at multiple places we build on the techniques of [13] and [2] which allow us to deal with maximal end components (sometimes called strongly communicating sets) of an MDP separately, we often need to extend these techniques. Unlike the works [13] and [2] which study multiple "independent" objectives, in the case of the global variance any change of value in the expected mean payoff implies a change of value of the variance.…”

“…Then the user gets 2 Mbits/sec connection almost surely, but since the individual runs are apparently "unstable", he may still see a lot of stuttering in the video stream. As an appropriate measure for the stability of individual runs, we propose local variance, which is defined as the long-run average of (r i (ω) − mp(ω)) 2 , where r i (ω) is the reward of the i-th action executed in a run ω and mp(ω) is the mean payoff of ω. Hence, local variance says how much the rewards of the actions executed along a given run deviate from the mean payoff of the run on average.…”