Bilinear value networks

Abstract

The dominant framework for off-policy multi-goal reinforcement learning involves estimating goal conditioned Q-value function. When learning to achieve multiple goals, data efficiency is intimately connected with generalization of the Q-function to new goals. The de-facto paradigm is to approximate Q(s, a, g) using monolithic neural networks. To improve generalization of the Q-function, we propose a bilinear decomposition that represents the Q-value via a low-rank approximation in the form of a dot product between two vector fields. The first vector field, f (s, a), captures the environment's local dynamics at the state s; whereas the second component, φ(s, g), captures the global relationship between the current state and the goal. We show that our bilinear decomposition scheme substantially improves data efficiency, and has superior transfer to out-of-distribution goals compared to prior methods. Empirical evidence is provided on the simulated Fetch robot task-suite, and dexterous manipulation with a Shadow hand.

Metric Residual Networks for Sample Efficient Goal-Conditioned Reinforcement Learning

“…Q(s, a, g) = f (s, a) φ(g). The motivation behind bilinear value networks is that decomposing Q(s, a, g) = f (s, a) φ(s, g) may result in better learning efficiency compared to the low-rank bilinear decomposition (Hong, Yang, and Agrawal 2022). Besides the above approaches designed specifically for GCRL, Pitis et al (2020) proposed the Deep Norm (DN) and Wide Norm (WN) families of neural networks that respect the triangle inequality.…”

“…2.1) into a bilinear network, e.g. either Q(s, a, g) = f (s, a) φ(g) (Schaul et al 2015) or Q(s, a, g) = f (s, a) φ(s, g) (Hong, Yang, and Agrawal 2022), where f and g are separate neural modules. The principle behind these designs is to inject useful inductive bias into the architecture.…”

“…they are restricted to only approximate norm-induced quasi-metrics. On the other hand, the conventional monolithic multi-layer perceptron (MLP) modeling Q(s, a, g), and the recently proposed bilinear value network (BVN)(Hong, Yang, and Agrawal 2022) both inherit the universal approximation guarantee from general neural networks, but they do not enforce the triangle inequality, thus making them less computationally efficient to learn. The Poisson Quasi-metric Embedding (PQE) (Wang and Isola 2022) enjoys the same theoretical guarantee as MRN.…”

“…Success rate over training epochs for MRN (ours), the monolithic network, BVN(Hong, Yang, and Agrawal 2022), DN and WN(Pitis et al 2020), and PQE (Wang and Isola 2022), on 12 GCRL environments from(Plappert et al 2018). Ablation study on individual symmetric/asymmetric parts of MRN, and on feeding all (s, a, g) to e 2 .…”

Metric Residual Network for Sample Efficient Goal-Conditioned Reinforcement Learning

“…For actor-criticlike methods, prior work has proposed decomposing the critic function (a.k.a the action-value function Q(s, a, g), see Sec. ) into a bilinear network, e.g., either Q(s, a, g) = f (s, a) ⊤ ϕ(g) (Schaul et al 2015) or Q(s, a, g) = f (s, a) ⊤ ϕ(s, g) (Hong, Yang, and Agrawal 2022), where f and g are separate neural modules. The principle behind these designs is to inject useful inductive bias into the architecture.…”