Macro Action Selection with Deep Reinforcement Learning in StarCraft

Abstract

StarCraft (SC) is one of the most popular and successful Real Time Strategy (RTS) games. In recent years, SC is also widely accepted as a challenging testbed for AI research because of its enormous state space, partially observed information, multi-agent collaboration, and so on. With the help of annual AIIDE and CIG competitions, a growing number of SC bots are proposed and continuously improved. However, a large gap remains between the top-level bot and the professional human player. One vital reason is that current SC bots mainly rely on predefined rules to select macro actions during their games. These rules are not scalable and efficient enough to cope with the enormous yet partially observed state space in the game. In this paper, we propose a deep reinforcement learning (DRL) framework to improve the selection of macro actions. Our framework is based on the combination of the Ape-X DQN and the Long-Short-Term-Memory (LSTM). We use this framework to build our bot, named as LastOrder. Our evaluation, based on training against all bots from the AIIDE 2017 StarCraft AI competition set, shows that LastOrder achieves an 83% winning rate, outperforming 26 bots in total 28 entrants.

A Mean-VaR Based Deep Reinforcement Learning Framework for Practical Algorithmic Trading

“…The Ape-x algorithm separates data collection from strategy learning, uses multiple parallel agents to collect experience data, shares a large experience data buffer, and sends it to learners for learning. The original Ape-X, which was based on DQN and Deep Deterministic Policy Gradient(DDPG), was utilized in the feedback flow separation control system [53], StarCraft games [54] and controlling vehicles for autonomous driving [55]. The following two characteristics are where this paper most clearly improves: we first connect the Ape-x with a distributed reinforcement learning framework for off-policy learning.…”