Convex duality for stochastic shortest path problems in known and unknown environments

Abstract

This paper studies Stochastic Shortest Path (SSP) problems in known and unknown environments from the perspective of convex optimisation. It first recalls results in the known parameter case, and develops understanding through different proofs. It then focuses on the unknown parameter case, where it studies extended value iteration (EVI) operators. This includes the existing operators used in Rosenberg et al. [25] and Tarbouriech et al. [30] based on the 1 norm and supremum norm, as well as defining EVI operators corresponding to other norms and divergences, such as the KL-divergence. This paper shows in general how the EVI operators relate to convex programs, and the form of their dual, where strong duality is exhibited.This paper then focuses on whether the bounds from finite horizon research of Neu and Pike-Burke [20] can be applied to these extended value iteration operators in the SSP setting. It shows that similar bounds to [20] for these operators exist, however they lead to operators that are not in general monotone and have more complex convergence properties. In a special case we observe oscillating behaviour. This paper generates open questions on how research may progress, with several examples that require further examination.