Benchmarking for Bayesian Reinforcement Learning

Abstract

In the Bayesian Reinforcement Learning (BRL) setting, agents try to maximise the collected rewards while interacting with their environment while using some prior knowledge that is accessed beforehand. Many BRL algorithms have already been proposed, but the benchmarks used to compare them are only relevant for specific cases. The paper addresses this problem, and provides a new BRL comparison methodology along with the corresponding open source library. In this methodology, a comparison criterion that measures the performance of algorithms on large sets of Markov Decision Processes (MDPs) drawn from some probability distributions is defined. In order to enable the comparison of non-anytime algorithms, our methodology also includes a detailed analysis of the computation time requirement of each algorithm. Our library is released with all source code and documentation: it includes three test problems, each of which has two different prior distributions, and seven state-of-the-art RL algorithms. Finally, our library is illustrated by comparing all the available algorithms and the results are discussed.

Reinforcement Learning for Electric Power System Decision and Control: Past Considerations and Perspectives

“…In particular, let us mention Bayesian RL approaches (see Ghavamzadeh et al (2015) for an extensive literature review), which offer two interesting features: by assuming a prior distribution on potential (unknown) environments, Bayesian RL (i) allows to formalize Bayesian-optimal exploration / exploitation strategies, and (ii) offers the opportunity to incorporate prior knowledge into the prior distribution. However, most Bayesian RL algorithms suffer computational complexity (Castronovo et al (2016)). …”

Approximate Bayes Optimal Policy Search using Neural Networks

“…The accuracy is depending on the number of nodes those algorithms are able to visit, which is limited by an on-line computation time budget. Despite theoretical guarantees to reach Bayesian optimality offered by BL approaches 1 , they may not be applicable when the time budget that can be allocated for on-line decision making is short (Castronovo et al, 2015). Another method, Smarter Best of Sampled Set (SBOSS) (Castro and Precup, 2010), samples several MDPs from the posterior distribution, builds a merged MDP, and computes its Q-function.…”

“…In practice, this happens for example when training a drone to fly in a safe environment before sending it on the operation field (Zhang et al, 2015). This is called offline training and can be beneficial to the online performance in the real environment, even if prior knowledge is inaccurate (Castronovo et al, 2014).…”

“…Instead, they rely on Bayes updates and sampling techniques during the interaction, which may be too computationally expensive, even on very small MDPs (Castronovo et al, 2015). In order to reduce significantly this cost, we propose a new practical algorithm to solve BAMDPs: Artificial Neural Networks for Bayesian Reinforcement Learning (ANN-BRL).…”

“…In our experiments, we used a benchmark recently introduced in (Castronovo et al, 2015). It compares all the major state-of-the-art BRL algorithms on a wide array of test problems, and provides a detailed computation time analysis.…”