Modular Multi-Objective Deep Reinforcement Learning with Decision Values

Abstract

In this work we present a method for using Deep Q-Networks (DQNs) in multi-objective environments. Deep Q-Networks provide remarkable performance in single objective problems learning from high-level visual state representations. However, in many scenarios (e.g in robotics, games), the agent needs to pursue multiple objectives simultaneously. We propose an architecture in which separate DQNs are used to control the agent's behaviour with respect to particular objectives. In this architecture we introduce decision values to improve the scalarization of multiple DQNs into a single action. Our architecture enables the decomposition of the agent's behaviour into controllable and replaceable sub-behaviours learned by distinct modules. Moreover, it allows to change the priorities of particular objectives post-learning, while preserving the overall performance of the agent. To evaluate our solution we used a game-like simulator in which an agentprovided with high-level visual input -pursues multiple objectives in a 2D world.

Efficient Multi-objective Reinforcement Learning via Multiple-gradient Descent with Iteratively Discovered Weight-Vector Sets

“…Typical multi-objective optimization (MOO) studies the problem of optimizing a set of possibly conflicting objectives. One main strategy to solve this problem is the scalarization approach (Ward & Lee, 2001;Nguyen, 2018;Tajmajer, 2018;Vamplew et al, 2017;Van Moffaert et al, 2013), where one or several single-objective optimization problems are solved. Two major disadvantages of these approaches are (1) the choice of the weighting factors is needed, leading to the burden of choosing them in the model, and (2) scalarization only results in a properly efficient solution (Ward & Lee, 2001).…”

“…In the multi-objective setting, the completion of a task requires the simultaneous satisfaction of multiple objectives such as balancing power consumption and performance in Web servers (Tesauro, et al, 2008). Such problems can be modeled as multi-objective Markov decision processes (MOMDPs) and solved by some existing multi-objective reinforcement learning (MORL) algorithms (Tesauro et al, 2008;Nguyen, 2018;Tajmajer, 2018;Abels, et al, 2018;Van Moffaert, Drugan, & Nowé, 2013;Vamplew, Dazeley, & Foale, 2017) based on the assumption that either the weighting factor for different objective functions or the ordering information is available. The solution for parameterized algorithms assumes the weighting factor for different objective functions can be obtained either directly (i.e., known a priori ) or indirectly (through learning).…”

“…For example, an approach was proposed to use both linear weighted sum and nonlinear thresholded lexicographic ordering methods to develop a multi-objective deep RL framework that includes both single-and multi-policy strategies (Nguyen, 2018). An architecture that uses separated deep Q-networks (DQNs) was proposed to control the agent's behavior with respect to particular objectives (Tajmajer, 2018). Each DQN has an additional decision value output that acts as a dynamic weight that is used while summing up Q-values.…”

Relationship Explainable Multi-objective Reinforcement Learning with Semantic Explainability Generation

“…The author in [15] proposed the use of both linear weighted sum and nonlinear thresholded lexicographic ordering methods to develop a multi-objective deep RL framework that includes both single-and multi-policy strategies. The author in [16] proposed an architecture in which separated deep Q-networks (DQNs) are used to control the agent's behavior with respect to particular objectives. Then, each DQN has an additional decision value output that acts as a dynamic weight used while summing up Q-values.…”

“…In summary, most of the algorithms are based on the scalarization method to transform the multi-objective problem into a single objective one. The scalarization can be nonlinear or linear [15], [16], [17], [18]. Other advanced methods include, e.g., the convex hull [19], the varying parameters approaches [20], the constraint method [21], the sequential method [22], and the max-min method [23].…”