Learning the structure of Factored Markov Decision Processes in reinforcement learning problems

Abstract

Recent decision-theoric planning algorithms are able to find optimal solutions in large problems, using Factored Markov Decision Processes (fmdps). However, these algorithms need a perfect knowledge of the structure of the problem. In this paper, we propose sdyna, a general framework for addressing large reinforcement learning problems by trial-and-error and with no initial knowledge of their structure. sdyna integrates incremental planning algorithms based on fmdps with supervised learning techniques building structured representations of the problem. We describe spiti, an instantiation of sdyna, that uses incremental decision tree induction to learn the structure of a problem combined with an incremental version of the Structured Value Iteration algorithm. We show that spiti can build a factored representation of a reinforcement learning problem and may improve the policy faster than tabular reinforcement learning algorithms by exploiting the generalization property of decision tree induction algorithms.

TeXDYNA: Hierarchical Reinforcement Learning in Factored MDPs

“…TeXDYNA ideas are first inspired by Sutton's DYNA architecture [8], enriched and adapted to FMDPs by [6]. Second, as to the exit oriented options representation, some ideas come from the HEXQ [4] and VISA [10,5] frameworks, where the exit definition proposed in HEXQ is extended to include variable change and context in order to address more complex structures.…”

“…Structured Dynamic Programming (SDP) algorithms such as SVI [7] make profit of this structure to compute a policy compactly. Structured-DYNA (SDYNA) [6] is a general framework that adapts indirect RL of the DYNA family [8] to the FMDP framework. SPITI is a particular instance of SDYNA based on a decision trees induction process to learn the structure of the problem and on SVI to obtain an efficient policy.…”

“…• action a ← current action a • variable change v ch ← value in the branch, value in the leaf 8 9 • context c ← variables of the branch that leads to the leaf l As to incrementality, [6] proposes to re-initialize the value function each time the reward function changes, mainly because the structure of the value function results from the reward function. In the incremental options learning case, the value function tree is reset each time the reward function changes in order to take into account every modification that changes the structure of the policy.…”

“…As to hierarchical representations, HEXQ [4] and VISA [5] are two algorithms designed to solve this problem. Besides, for FMDPs, [6] have proposed SDYNA to solve Factored Reinforcement Learning (FRL) problems. Here, we propose TeXDYNA, an algorithm that combines the benefits of HRL and FRL, building an HRL-augmented version of SDYNA. The paper is organized as follows.…”

Improving reinforcement learning by using sequence trees

“…Starting from the variable that changes most frequently, a level of hierarchy is generated for each variable, and at each level the states that are represented by the values of the corresponding variable are partitioned into regions; the regions are identified by state-action pairs (called exits) that cause unpredictable transitions; separate policies that leave each region through its exists form the temporal abstractions. In the factored reinforcement learning setting, TexDYNA algorithm of Kozlova et al (2009) simultaneously decomposes a factored MDP into a set of options and improves incrementally the local policy of each option by using a particular decision tree based instance of the model-based reinforcement learning framework SDYNA (Degris et al 2006). In their approach, the options are determined by exits as in HEXQ, but the variable whose values determine the context itself are explicit in the exit definition; furthermore, to ensure their relevance exits are updated every time the model of transitions change.…”