An analytic solution to discrete Bayesian reinforcement learning

Abstract

Reinforcement learning (RL) was originally proposed as a framework to allow agents to learn in an online fashion as they interact with their environment. Existing RL algorithms come short of achieving this goal because the amount of exploration required is often too costly and/or too time consuming for online learning. As a result, RL is mostly used for offline learning in simulated environments. We propose a new algorithm, called BEETLE, for effective online learning that is computationally efficient while minimizing the amount of exploration. We take a Bayesian model-based approach, framing RL as a partially observable Markov decision process. Our two main contributions are the analytical derivation that the optimal value function is the upper envelope of a set of multivariate polynomials, and an efficient pointbased value iteration algorithm that exploits this simple parameterization.

Regret Bounds for Reinforcement Learning with Policy Advice

“…1 shows per-step regret of the algorithms as the function of the number of states. As predicted by the theoretical bounds, the per-step regret ∆ of UCRL2 significantly increases as the number of states increases, whereas the average regret of our RLPA is essentially independent of the state space size 10 . Although UCWM has a lower regret than RLPA for a small number of states, it quickly loses its advantage as the number of states grows.…”

Sample Complexity Bounds of Exploration

“…In general, computing the Bayes optimal policy is often intractable, making approximation inevitable (see, e.g., Duff (2002)). It remains an active research area to develop efficient algorithms to approximate the Bayes optimal policy (Poupart et al, 2006;Kolter and Ng, 2009).…”

Real time targeted exploration in large domains

“…This augmented state representation results in a enormous state space, making the full Bayesian algorithm intractable. Attempts have been made to approximate the full algorithm by parameterizing the model and tying model parameters together [7] or sampling from the model distribution [8], [9], but these methods are still only tested in domains with 5-36 states. In addition to requiring a large amount of time to compute a policy, these methods must maintain a belief state over the model and require the user to create a well-defined model prior.…”