Unpredictable Planning Under Partial Observability

Abstract

We study the problem of synthesizing a controller that maximizes the entropy of a partially observable Markov decision process (POMDP) subject to a constraint on the expected total reward. Such a controller minimizes the predictability of a decision-maker's trajectories while guaranteeing the completion of a task expressed by a reward function. First, we prove that a decision-maker with perfect observations can randomize its paths at least as well as a decision-maker with partial observations. Then, focusing on finite-state controllers, we recast the entropy maximization problem as a so-called parameter synthesis problem for a parametric Markov chain (pMC). We show that the maximum entropy of a POMDP is lower bounded by the maximum entropy of this pMC. Finally, we present an algorithm, based on a nonlinear optimization problem, to synthesize an FSC that locally maximizes the entropy of a POMDP over FSCs with the same number of memory states. In numerical examples, we demonstrate the proposed algorithm on motion planning scenarios.

Smoother Entropy for Active State Trajectory Estimation and Obfuscation in POMDPs

“…Most recently, the problem of obfuscating entire state trajectories from any conceivable estimator has been investigated by drawing on ideas from privacy in static settings (e.g., datasets) including differential privacy [20], [34], [35] and information theory [20], [21], [36], [37]. These works, however, sidestep complete POMDP treatments either by only increasing the state's unpredictability [22], [25] or by only degrading the measurements [21], [36], [37] (rather than a combination of the two). Furthermore, as noted in [38], POMDPs for information-averse or obfuscation problems frequently involve cost and cost-to-go functions that are not concave in the belief state, and so have mostly been avoided until recently because no satisfying (approximate) solution techniques existed.…”

“…where c k , ˜ , and g are defined in ( 5), (25), and (28). Then, the active obfuscation problem (5) is equivalent to:…”

“…form to aid state estimation include active state estimation and dual control in automatic control [3]- [6], controlled sensing in signal processing and robotics [7]- [11], and active simultaneous localisation and mapping (SLAM) in robotics [12]- [17]. Conversely, applications in which the problem has been investigated in its active obfuscation form to hinder state estimation include privacy in cyber-physical systems [18]- [23], and covert navigation in robotics [24], [25]. Despite these many applications, few works have explicitly addressed active estimation or obfuscation of entire state trajectories, with most instead focusing on aiding or hindering state estimation as it relates to the performance of Bayesian filters.…”

Smoothing-Averse Control: Covertness and Privacy from Smoothers

“…a dynamical system) is particularly challenging since system inputs and/or outputs from isolated time instances have the potential to reveal information about the entire state trajectory through correlations introduced by the system dynamics. The design of both controllers [1]- [4], [9], [14] and output filters [8], [13] that limit the disclosure of dynamical system state information through inputs and/or outputs has therefore attracted considerable recent attention (see also [15], [16] and references therein). Despite these efforts, few works have addressed the problem of how best to control a system to conceal its entire state trajectory from an adversary that employs a Bayesian smoother for state trajectory estimation.…”

“…Whilst many works on covert motion planning have been developed in the context of robot navigation with deterministic descriptions of adversary sensing capabilities (e.g., sensor field of view) [21]- [23], covert motion planning is increasingly being considered with approaches inspired by information-theoretic privacy. For example, information-theoretic approaches that seek to make the future trajectory or policy of an agent difficult to predict and/or infer have been proposed in [3], [4], [24] on the basis of maximising the control policy entropy, and in [1] on the basis of Fisher information. One of the few approaches that explicitly seeks to conceal an agent's entire trajectory is proposed in [2] and involves minimising the total probability mass (over a specific time horizon) that the state estimates at individual times from a Bayesian filter have at the true states.…”