Approximate Information States for Worst-Case Control and Learning in Uncertain Systems

Abstract

In this paper, we investigate discrete-time decisionmaking problems in uncertain systems with partially observed states. We consider a non-stochastic model, where uncontrolled disturbances acting on the system take values in bounded sets with unknown distributions. We present a general framework for decision-making in such problems by developing the notions of information states and approximate information states. In our definition of an information state, we introduce conditions to identify for an uncertain variable sufficient to construct a dynamic program (DP) that computes an optimal strategy. We show that many information states from the literature on worstcase control actions, e.g., the conditional range, are examples of our more general definition. Next, we relax these conditions to define approximate information states using only output variables, which can be learned from output data without knowledge of system dynamics. We use this notion to formulate an approximate DP that yields a strategy with a bounded performance loss. Finally, we illustrate the application of our results in control and reinforcement learning using numerical examples.

Approximate Information States for Worst-case Control of Uncertain Systems

“…3) Hausdorff Distance: Consider two bounded, nonempty subsets X , Y of a metric space (S , η), where η : X × Y → R ≥0 is the metric. The Hausdorff distance between X and Y is the pseudo-metric [25, Chapter 1.12]: [17,Lemma 5]:…”

“…This proves (18) by induction. Then, (17) follows directly for all t ∈ N and all n ∈ N by substituting ( 18) into (9) and selecting the horizon T = t + n − 1.…”

“…Theorem 1 allows us to characterize the error between the optimal value V 0 (y 0 ) and Λ n (σ 0 (y 0 ), 1) for any y 0 ∈ Y by selecting t = 0 in (17), it follows that γ n •c min 1−γ + Λ n (σ 0 (y 0 ), 1) ≤ V 0 (y 0 ) ≤ Λ n (σ 0 (y 0 ), 1) + γ n •c max 1−γ (recall that a 0 = 0 and z 0 = 1). These bounds imply that as the number of iterations n → ∞, the fixed point Λ ∞ satisfies…”

“…While these formulations improve the performance under a distribution mismatch, many safetycritical applications require further guarantees on the worstcase performance of a strategy against either adversarial attacks or system failure, e.g., cyber-security [10] and cyberphysical networks [11]. A non-stochastic formulation is suit-able in such applications, where the agent has no knowledge of the distributions on the uncertainties and uses only their set of feasible values to compute a control strategy that minimizes the maximum possible cost [12]- [17]. This nonstochastic formulation is both maximally robust for a given set of uncertainties and most risk-averse under any feasible prior distribution [18].…”