Least Inferable Policies for Markov Decision Processes

Abstract

In a variety of applications, an agent's success depends on the knowledge that an adversarial observer has or can gather about the agent's decisions. It is therefore desirable for the agent to achieve a task while reducing the ability of an observer to infer the agent's policy. We consider the task of the agent as a reachability problem in a Markov decision process and study the synthesis of policies that minimize the observer's ability to infer the transition probabilities of the agent between the states of the Markov decision process. We introduce a metric that is based on the Fisher information as a proxy for the information leaked to the observer and using this metric formulate a problem that minimizes expected total information subject to the reachability constraint. We proceed to solve the problem using convex optimization methods. To verify the proposed method, we analyze the relationship between the expected total information and the estimation error of the observer, and show that, for a particular class of Markov decision processes, these two values are inversely proportional. M. O. Karabag is with the

Smoothing-Averse Control: Covertness and Privacy from Smoothers

“…Controlling or limiting the information that cyber-physical systems disclose during their operation is a challenging yet increasingly important problem across many application domains including robotics and autonomous vehicles [1]- [7], networked control [8], [9], smart grid and power systems [10]- [12], and sensor networks [13]. Concealing information about the state of a cyber-physical system with dynamics (i.e.…”

“…a dynamical system) is particularly challenging since system inputs and/or outputs from isolated time instances have the potential to reveal information about the entire state trajectory through correlations introduced by the system dynamics. The design of both controllers [1]- [4], [9], [14] and output filters [8], [13] that limit the disclosure of dynamical system state information through inputs and/or outputs has therefore attracted considerable recent attention (see also [15], [16] and references therein). Despite these efforts, few works have addressed the problem of how best to control a system to conceal its entire state trajectory from an adversary that employs a Bayesian smoother for state trajectory estimation.…”

“…Whilst many works on covert motion planning have been developed in the context of robot navigation with deterministic descriptions of adversary sensing capabilities (e.g., sensor field of view) [21]- [23], covert motion planning is increasingly being considered with approaches inspired by information-theoretic privacy. For example, information-theoretic approaches that seek to make the future trajectory or policy of an agent difficult to predict and/or infer have been proposed in [3], [4], [24] on the basis of maximising the control policy entropy, and in [1] on the basis of Fisher information. One of the few approaches that explicitly seeks to conceal an agent's entire trajectory is proposed in [2] and involves minimising the total probability mass (over a specific time horizon) that the state estimates at individual times from a Bayesian filter have at the true states.…”

“…[9], [10], [12]) or systems with discrete finite state spaces (cf. [1]- [4], [24]). Key to our reformulation of our smoothing-averse control problem as a fully observed stochastic optimal control solvable via dynamic programming is thus the development of a novel additive expression for the (joint) entropy of Bayesian smoother estimates for general nonlinear state-space models (which is potentially of independent interest).…”

Minimizing the Information Leakage Regarding High-Level Task Specifications

“…Recently, the works of Savas et al (2019) and Karabag et al (2019) focused on synthesizing policies that are either unpredictable or difficult for an adversarial observer to infer. These studies focused on the low-level actions rather than on the highlevel specifications as we do.…”