The Markovian Price of Information

Abstract

Suppose there are n Markov chains and we need to pay a per-step price to advance them. The "destination" states of the Markov chains contain rewards; however, we can only get rewards for a subset of them that satisfy a combinatorial constraint, e.g., at most k of them, or they are acyclic in an underlying graph. What strategy should we choose to advance the Markov chains if our goal is to maximize the total reward minus the total price that we pay? In this paper we introduce a Markovian price of information model to capture settings such as the above, where the input parameters of a combinatorial optimization problem are given via Markov chains. We design optimal/approximation algorithms that jointly optimize the value of the combinatorial problem and the total paid price. We also study robustness of our algorithms to the distribution parameters and how to handle the commitment constraint. Our work brings together two classical lines of investigation: getting optimal strategies for Markovian multi-armed bandits, and getting exact and approximation algorithms for discrete optimization problems using combinatorial as well as linear-programming relaxation ideas.

Multi-token Markov Game with Switching Costs

“…Recent works [KWW16,Sin18] studied the problem under richer combinatorial constraints. Gupta et al [GJSS19] study a more general problem: there is a given packing constraint F ⊆ 2 [𝑛] (e.g., matroid, matching, knapsack) of subsets of chains. e goal is to make a subset 𝑆 of chains to reach their targets (𝑆 ∈ F ), while minimizing the dis-utility (with upward-closed constraint) or maximize the utility (with downward-closed constraint).…”

“…Now we de ne a prevailing cost [DTW03] and an epoch [GJSS19]. A trajectory is a sequence of states in a Markov system traversed by a player.…”

Spatial Privacy Pricing

“…Singla [16] also studied the problem of maximizing the buyer's profit when information is available at a price, but also can only be purchased at full price, which can be considered as a simpler form of our problem and is used as a baseline in our experiments. In [9], Gupta et. al extends on the capability of purchasing the same data multiple times.…”

“…In the first phase, the buyer would buy all available data points in a large bounding box around the target region at a small starting price 0 (lines [2][3][4][5][6][7][8][9]). Then in the second phase (lines ), the buyer repeatedly calculates EIPs (lines [10][11][12][13]), takes a buying action based on the high potential EIPs (line [15,27]) and uses the newly purchased noisy data, if any, to guide the next action (line [19,29]).…”

“…Other studies have attempted to find adaptive solutions to similar problems. However, they either considered a simpler version of this problem [16] or had different objectives [6,8,9,14], thus, resulting in inapplicable solutions. Finally, the approaches for location privacy protection such as spatial cloaking [18], differential privacy [5] or Geo-Indistinguishablity [3] are relevant but orthogonal to our work as we discuss in Section 2.2.…”