Efficient Multi-objective Reinforcement Learning via Multiple-gradient Descent with Iteratively Discovered Weight-Vector Sets

Abstract


 
 
 Solving multi-objective optimization problems is important in various applications where users are interested in obtaining optimal policies subject to multiple (yet often conflicting) objectives. A typical approach to obtain the optimal policies is to first construct a loss function based on the scalarization of individual objectives and then derive optimal policies that minimize the scalarized loss function. Albeit simple and efficient, the typical approach provides no insights/mechanisms on the optimization of multiple objectives due to the lack of ability to quantify the inter-objective relationship. To address the issue, we propose to develop a new efficient gradient-based multi-objective reinforcement learning approach that seeks to iteratively uncover the quantitative inter-objective relationship via finding a minimum-norm point in the convex hull of the set of multiple policy gradients when the impact of one objective on others is unknown a priori. In particular, we first propose a new PAOLS algorithm that integrates pruning and approximate optimistic linear support algorithm to efficiently discover the weight-vector sets of multiple gradients that quantify the inter-objective relationship. Then we construct an actor and a multi-objective critic that can co-learn the policy and the multi-objective vector value function. Finally, the weight discovery process and the policy and vector value function learning process can be iteratively executed to yield stable weight-vector sets and policies. To validate the effectiveness of the proposed approach, we present a quantitative evaluation of the approach based on three case studies.
 
 

Off-Policy Meta-Reinforcement Learning With Belief-Based Task Inference

“…Reward vectors. The existing methods for MORL (Roijers et al, 2014;Mossalam et al, 2016;Xu et al, 2020;Cao & Zhan, 2021) and succussor features (SF) (Barreto et al, 2017;Borsa et al, 2018;Hunt et al, 2019;Barreto et al, 2019;Zahavy et al, 2021) are related in terms of using reward vectors. In conventional SF settings, the agent optimizes its policy under condition that scalar rewards are given.…”

Multi-objective reinforcement learning based on nonlinear scalarization and long-short-term optimization

“…Hu et al combined Lamarckian local search and deep multi-objective reinforcement learning to dispatch water valves and fire hydrants to isolate contaminated water and reduce residual concentrations of contaminants in water distribution networks (Hu et al , 2022). Cao and Zhan proposed a new and efficient gradient-based multi-objective reinforcement learning method, which aimed to iteratively reveal the relationship between objectives by finding the minimum norm point in the convex hull of multiple policy gradient sets (Cao and Zhan, 2021). Hu et al proposed the MO-MIX algorithm to solve the multi-objective multi-agent reinforcement learning problem based on the centralized training and decentralized execution framework (Hu et al , 2023).…”