On learning history based policies for controlling Markov decision processes

Abstract

Reinforcement learning (RL) folklore suggests that history-based function approximation methods, such as recurrent neural nets or history-based state abstraction, perform be er than their memory-less counterparts, due to the fact that function approximation in Markov decision processes (MDP) can be viewed as inducing a Partially observable MDP. However, there has been li le formal analysis of such history-based algorithms, as most existing frameworks focus exclusively on memory-less features. In this paper, we introduce a theoretical framework for studying the behaviour of RL algorithms that learn to control an MDP using history-based feature abstraction mappings. Furthermore, we use this framework to design a practical RL algorithm and we numerically evaluate its e ectiveness on a set of continuous control tasks.

Approximate Information States for Worst-Case Control and Learning in Uncertain Systems

“…In the stochastic formulation, the notion of approximate information states was presented in [81] to address the challenges of control and learning with partial observations. Approximate information states can improve the computational tractability of control problems with large state spaces at the cost of a bounded loss in performance [82]. The explicit performance bounds of a finite-memory based approximate information state were derived in [83].…”