Markov Chains With Maximum Return Time Entropy for Robotic Surveillance

Abstract

Motivated by robotic surveillance applications, this paper studies the novel problem of maximizing the return time entropy of a Markov chain, subject to a graph topology with travel times and stationary distribution. The return time entropy is the weighted average, over all graph nodes, of the entropy of the first return times of the Markov chain; this objective function is a function series that does not admit in general a closed form.The paper features theoretical and computational contributions. First, we obtain a discrete-time delayed linear system for the return time probability distribution and establish its convergence properties. We show that the objective function is continuous over a compact set and therefore admits a global maximum; a unique globally-optimal solution is known only for complete graphs with unitary travel times. We then establish upper and lower bounds between the return time entropy and the well-known entropy rate of the Markov chain. To compute the optimal Markov chain numerically, we establish the asymptotic equality between entropy, conditional entropy and truncated entropy, and propose an iteration to compute the gradient of the truncated entropy. Finally, we apply these results to the robotic surveillance problem. Our numerical results show that, for a model of rational intruder over prototypical graph topologies and test cases, the maximum return time entropy chain performs better than several existing Markov chains. arXiv:1803.07705v2 [math.OC]

Robotic Surveillance Based on the Meeting Time of Random Walks

“…An interesting extension of the work discussed here would be to consider walkers moving with travel times similar to the cases studied in [14] and [25].…”

“…Stochastic vehicle routing strategies have the desirable property that an intruder cannot predictably plan a path to avoid surveillance agents. The authors in [25], [16], [14] use Markov chains to design surveillance strategies. A novel convex optimization formulation is used to design strategies with minimum mean hitting time in [25].…”

Entropy Maximization for Markov Decision Processes Under Temporal Logic Constraints

“…In [19], [20], the authors study the problem of synthesizing a transition matrix with maximum entropy for an irreducible MC subject to graph constraints. The problem studied in this paper is considerably different from that problem since MDPs represent a more general model than MCs, and an MC induced from an MDP by a policy is not necessarily irreducible.…”

Markov Chain-Based Stochastic Strategies for Robotic Surveillance

“…To obtain maximally unpredictable surveillance strategies, the authors in [22] design Markov chains with maximum entropy rate: numerous properties of the maxentropic Markov chain are established and a fast algorithm to compute the optimal strategy is proposed. A new notion that quantifies unpredictability of surveillance strategies through the entropy in return time distributions has been recently introduced and characterized in [18]. This new concept is particularly relevant and useful in cases when only local observations are available to potential intruders.…”