Cache-Efficient Memory Representation of Markov Decision Processes

Abstract

Research in automated planning typically focuses on the development of new or improved algorithms. Yet, an equally important but often overlooked topic is that of how to actually implement these algorithms eciently. In this study, we are making an attempt to close this gap in the context of optimal Markov Decision Process (MDP) planning. Precisely, we present a novel cache-ecient memory representation of MDPs, which we call CSR-MDP, that takes advantage of low-level hardware features such as memory hierarchy. We evaluate the speed improvement provided by our memory representation by comparing the performance of CSR-MDP with the performance obtained by traditional MDP representation. We show that by using our CSR-MDP memory representation, existing MDP solvers, including VI, LRTDP and TVI, are able to nd an optimal policy an order of magnitude faster.

Cache-Efficient Dynamic Programming MDP Solver

“…There are two different ways to do it: (1) by changing the data-structures used to store the MDP in memory, and (2) by changing the algorithms per se. To the best of our knowledge, the former way has only been considered in one study [6] that proposed to store MDP instances in a new data-structure, called CSR-MDP, inspired by the compressed-sparse-row representation of graphs. This CSR-MDP memory representation consists of five arrays (S, C, A, N, P ), where S contains the states' actions ids; C contains the cost of each action; A contains the actions' effects ids; N and P contain respectively the effects' state transitions and probabilities.…”