Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation

Abstract

Reflecting on the last few years, the biggest breakthroughs in deep reinforcement learning (RL) have been in the discrete action domain. Robotic manipulation, however, is inherently a continuous control environment, but these continuous control reinforcement learning algorithms often depend on actor-critic methods that are sample-inefficient and inherently difficult to train, due to the joint optimisation of the actor and critic. To that end, we explore how we can bring the stability of discrete action RL algorithms to the robot manipulation domain. We extend the recently released ARM algorithm, by replacing the continuous next-best pose agent with a discrete next-best pose agent. Discretisation of rotation is trivial given its bounded nature, while translation is inherently unbounded, making discretisation difficult. We formulate the translation prediction as the voxel prediction problem by discretising the 3D space; however, voxelisation of a large workspace is memory intensive and would not work with a high density of voxels, crucial to obtaining the resolution needed for robotic manipulation. We therefore propose to apply this voxel prediction in a coarse-to-fine manner by gradually increasing the resolution. In each step, we extract the highest valued voxel as the predicted location, which is then used as the centre of the higher-resolution voxelisation in the next step. This coarse-to-fine prediction is applied over several steps, giving a near-lossless prediction of the translation. We show that our new coarse-to-fine algorithm is able to accomplish RLBench tasks much more efficiently than the continuous control equivalent, and even train some real-world tasks, tabular rasa, in less than 7 minutes, with only 3 demonstrations. Moreover, we show that by moving to a voxel representation, we are able to easily incorporate observations from multiple cameras. Videos and code found at: https://sites.google.com/ view/c2f-q-attention.

Bingham Policy Parameterization for 3D Rotations in Reinforcement Learning

“…Continuous control RL policies are commonly parameterized as Gaussians with diagonal covariance matrices [31,30,11], though other parameterizations have been considered, including Gaussians with covariance matrix via the Cholesky factor [1], Gaussian mixtures [38], Beta distributions [3], and Bernoulli distributions [32]. Rather than directly outputting continuous values, an alternative way of parameterizing a continuous control policy is via discretization, whether through growing action spaces [4] or coarse-to-fine networks [16]. All of these works share a common goal of moving away from the conventional Gaussian parameterization, but none are ideal when faced with an action space that requires rotation predictions.…”

“…Deep reinforcement learning (RL) is now actively used in many areas, including playing games [25,33], robot manipulation [24,16], and legged robotics [13,29]. The leading (general-purpose) algorithms within the continuous control RL community are either deterministic, such as DDPG [23] and TD3 [5], or stochastic, such as SAC [11] and PPO [31].…”