Optimized look‐ahead tree policies: a bridge between look‐ahead tree policies and direct policy search

Abstract

Direct policy search (DPS) and look-ahead tree (LT) policies are two widely used classes of techniques to produce high performance policies for sequential decision-making problems. To make DPS approaches work well, one crucial issue is to select an appropriate space of parameterized policies with respect to the targeted problem. A fundamental issue in LT approaches is that, to take good decisions, such policies must develop very large look-ahead trees which may require excessive online computational resources. In this paper, we propose a new hybrid policy learning scheme that lies at the intersection of DPS and LT, in which the policy is an algorithm that develops a small look-ahead tree in a directed way, guided by a node scoring function that is learned through DPS. The LT-based representation is shown to be a versatile way of representing policies in a DPS scheme, while at the same time, DPS enables to significantly reduce the size of the look-ahead trees that are required to take high-quality decisions.We experimentally compare our method with two other state-of-the-art DPS techniques and four common LT policies on four benchmark domains and show that it combines the advantages of the two techniques from which it originates. In particular, we show that our method: (1) produces overall better performing policies than both pure DPS and pure LT policies, (2) requires a substantially smaller number of policy evaluations than other DPS techniques, (3) is easy to tune and (4) results in policies that are quite robust with respect to perturbations of the initial conditions.

Optimized look-ahead trees: Extensions to large and continuous action spaces

“…However, OLT policies differ from LT ones in one significant way: whereas LT uses a generic node expansion heuristic, OLT relies on a parameterized node expansion heuristic exp score(n; θ) where the parameters θ are specifically optimized in an offline learning phase for the given target domain (f, ̺). The main advantage of OLT over LT is that this optimization can lead to a substantial reduction of the number of node expansions necessary to output good control actions (as was empirically demonstrated in [7], [9]), meaning that OLT can achieve the same performance as LT at a significantly lower online cost. (The disadvantage of OLT is of course that it needs this prior offline learning.)…”

“…Having made this distinction, we can characterize DPS and LT as lying at opposite ends of the offline complexity / online complexity spectrum [7]. DPS techniques typically require huge offline resources for two reasons.…”

“…OLT policies as described in [7], [9] are also intended for deterministic problems with finite and small action spaces, with the nodes being also defined over states. However, OLT policies differ from LT ones in one significant way: whereas LT uses a generic node expansion heuristic, OLT relies on a parameterized node expansion heuristic exp score(n; θ) where the parameters θ are specifically optimized in an offline learning phase for the given target domain (f, ̺).…”

“…The particular technique (or better, family of techniques) we are going to study is called optimized look-ahead tree policies (OLT) [7], [9]. OLT is a model-based hybrid technique that bridges the gap between look-ahead tree policies (LT) [3], [6] and direct policy search (DPS).…”

“…OLT was first introduced in [9], with the more comprehensive study in [7] forming the basis of the present work. Here, we extend the earlier approach in two ways:…”