High-Dimensional Function Approximation for Knowledge-Free Reinforcement Learning

Abstract

SZ-Tetris, a restricted version of Tetris, is a difficult reinforcement learning task. Previous research showed that, similarly to the original Tetris, value function-based methods such as temporal difference learning, do not work well for SZ-Tetris. The best performance in this game was achieved by employing direct policy search techniques, in particular the cross-entropy method in combination with handcrafted features. Nonetheless, a simple heuristic hand-coded player scores even higher. Here we show that it is possible to equal its performance with CMA-ES (Covariance Matrix Adaptation Evolution Strategy). We demonstrate that further improvement is possible by employing systematic n-tuple network, a knowledge-free function approximator, and VD-CMA-ES, a linear variant of CMA-ES for high dimension optimization. Last but not least, we show that a large systematic n-tuple network (involving more than 4 million parameters) allows the classical temporal difference learning algorithm to obtain similar average performance to VD-CMA-ES, but at 20 times lower computational expense, leading to the best policy for SZ-Tetris known to date. These results enrich the current understanding of difficulty of SZ-Tetris, and shed new light on the capabilities of particular search paradigms when applied to representations of various characteristics and dimensionality.

Mastering 2048 With Delayed Temporal Coherence Learning, Multistage Weight Promotion, Redundant Encoding, and Carousel Shaping

“…In this context, it is worth to emphasize the role of the (systematic) n-tuple networks for the overall result. Although they do not scale up to state spaces with large dimensionality [13], they have undeniable advantages for state spaces of moderate sizes and dimensions. They provide nonlinear transformations and computational efficiency.…”

“…The roots of reinforcement learning [14] in games can be tracked down to Samuel's famous work on Checkers [25], but it was not popularized until the Tesauro's TD-Gammon, a master-level program for Backgammon [34] obtained by temporal difference learning. Various reinforcement learning algorithms were applied with success to other games such as Othello [6], [15], [32], Connect 4 [37], Tetris [27], [13], or Atari video games [4]. Recently, AlphaGo used reinforcement learning among others to determine weights for a deep artificial neural network to beat a professional Go player [28].…”

“…1) n-Tuple Network: One particular type of function approximator is n-tuple network [15], which has been recently successfully applied to board games such as Othello [7], [17], [33], [12], Connect 4 [37], or Tetris [13].…”

Final Adaptation Reinforcement Learning for N-Player Games

“…We conclude that the part propagating the final reward of the other player back to the other player's previous state is vitally important. 7 If we analyze the no-FARL-agent we find that it has only 0.9% active weights while the good-working TD-FARL agent has 8% active weights. This comes because the other player (that is the one who loses the game since the current player created a winning state) has never the negative reward propagated back to previous states of that other player.…”

Discovering Rubik's Cube Subgroups using Coevolutionary GP

“…policies do not adapt while interacting with the application. Most evolutionary methods take this form, with neuroevolutionary algorithms such as CoSyNE [8], NEAT [24] or CMA-ES (for weight optimization) [10] representing specific examples.…”