Scaling All-Goals Updates in Reinforcement Learning Using Convolutional Neural Networks

Abstract

Being able to reach any desired location in the environment can be a valuable asset for an agent. Learning a policy to navigate between all pairs of states individually is often not feasible. An all-goals updating algorithm uses each transition to learn Q-values towards all goals simultaneously and off-policy. However the expensive numerous updates in parallel limited the approach to small tabular cases so far. To tackle this problem we propose to use convolutional network architectures to generate Q-values and updates for a large number of goals at once. We demonstrate the accuracy and generalization qualities of the proposed method on randomly generated mazes and Sokoban puzzles. In the case of on-screen goal coordinates the resulting mapping from frames to distance-maps directly informs the agent about which places are reachable and in how many steps. As an example of application we show that replacing the random actions in ε-greedy exploration by several actions towards feasible goals generates better exploratory trajectories on Montezuma's Revenge and Super Mario All-Stars games.

Partial Consistency for Stabilizing Undiscounted Reinforcement Learning

“…However, these algorithms require a complete evaluation before each update of policies, so they fail to achieve a high sampling efficiency. Cao et al [16] and Pardo et al [23] directly optimized the undiscounted return by augmenting the state space with the remaining time. Then, acyclic MDPs were constructed to eliminate transient traps.…”

“…For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/ for preserving the optimal policy [13], [16], [17], [22]- [24]. On the other hand, similar to the cases with undiscounted returns [13], [16], [17], [23], [25], γ close to 1 (e.g., γ > 0.99) also causes the training instability problem. Thus, the analysis of the instability of undiscounted RL helps to alleviate the instability of optimizing a discounted return with a large γ .…”

Offline Deep Reinforcement Learning for Dynamic Pricing of Consumer Credit

“…Lu et al [23] explored the use of causal models of the state dynamics for counterfactual data augmentation. Time limits in RL have been employed to manage task complexity and facilitate learning, with Pardo et al [28] analyzing their utility in diversifying training experience and boosting performance when combined with agent time-awareness.…”

Goal-Aware Cross-Entropy for Multi-Target Reinforcement Learning

“…The terms targets and goals have been used in diverse manners in the existing literature. There are subgoal generation methods that generate intermediate goals such as imagined goal [31,14] or random goals [34] to help agents solve the tasks. Multi-goal RL [47,35,13,10] aims to deal with multiple tasks, learning to reach different goal states for each task.…”