Q-Attention: Enabling Efficient Learning for Vision-Based Robotic Manipulation

Bingham Policy Parameterization for 3D Rotations in Reinforcement Learning

“…where future rewards are weighted with respect to the discount factor γ ∈ [0, 1). In this paper, we apply our Bingham Policy Parameterization (BPP) to three different algorithms: Soft-actor critic (SAC) [11], proximal policy optimization (PPO) [31], and attention-driven robotic manipulation (ARM) [14]. We briefly outline these below.…”

“…These pixel locations are used to crop the RGB image and organised point cloud inputs and thus drastically reduce the input size to the next stage of the pipeline; this next stage is an actor-critic next-best pose agent using SAC as the underlying algorithm. For further details on keypoint detection and demo augmentation, we point the reader to [14].…”

“…Finally, we move beyond the two toy-problems presented in the previous two sections, and show how our method scales in a number of ways: (1) to vision-based tasks, (2) to sparse rewarded tasks, and (3) to a more sophisticated robot learning algorithm. For these experiments, we use the same 8 RLBench tasks as used in James and Davison [14]. Note, unlike the previous section, we do not modify them to have shaped rewards or low-dimensional observations (i.e.…”

“…RLBench in its challenging default state); making the environment significantly harder. Given that PPO and SAC are unable to perform well in this domain (due to increased challenge of sparse rewards and image observations), we instead make use of ARM [14]. ARM is a state-of-the-art manipulation algorithm that introduced several core concepts which facilitate the learning of robot manipulation tasks given only a small number of demonstrations; these include Qattention, keyframe discovery, and demo augmentation.…”

“…One such example is an environment that requires a 3D rotation action space. Quaternions are often used as the output rotation representation when using deep networks [37,35,14], however due to the complex bi-modal and symmetric property, sampling a quaternion from a Gaussian does not seem appropriate.…”