Using Reinforcement Learning for Multi-policy Optimization in Decentralized Autonomic Systems – An Experimental Evaluation

Abstract

Abstract. Large-scale autonomic systems are required to self-optimize with respect to high-level policies, that can differ in terms of their priority, as well as their spatial and temporal scope. Decentralized multiagent systems represent one approach to implementing the required selfoptimization capabilities. However, the presence of multiple heterogeneous policies leads to heterogeneity of the agents that implement them. In this paper we evaluate the use of Reinforcement Learning techniques to support the self-optimization of heterogeneous agents towards multiple policies in decentralized systems. We evaluate these techniques in an Urban Traffic Control simulation and compare two approaches to supporting multiple policies. Our results suggest that approaches based on W-learning, which learn separately for each policy and then select between nominated actions based on current action importance, perform better than combining policies into a single learning process over a single state space. The results also indicate that explicitly supporting multiple policies simultaneously can improve waiting times over policies dedicated to optimizing for a single vehicle type.

Distributed W-Learning: Multi-Policy Optimization in Self-Organizing Systems

“…Dependency between policies has been confirmed in our previous work with non-collaborative agents [10]. We observed that implementing a single policy that only addresses emergency vehicles creates a backlog of other vehicles in the system, which in turn prevent emergency vehicles from proceeding.…”

“…From our previous experiments [10] we have observed that policy deployments can benefit from combining policies using W-learning independently at each agent. We have also observed a high dependency between policy deployments.…”

“…A number of multi-policy extensions have been implemented, as agents often have to simultaneously meet multiple policies. We reviewed some of those techniques in our previous work [10] and evaluated two approaches: combining policies' state spaces into a single learning process [14] and W-Learning [7]. W-learning was shown to yield better results in the scenarios evaluated, and to be suitable as a multi-policy optimization technique for UTC [10] when deployed on noncooperating agents.…”

“…We reviewed some of those techniques in our previous work [10] and evaluated two approaches: combining policies' state spaces into a single learning process [14] and W-Learning [7]. W-learning was shown to yield better results in the scenarios evaluated, and to be suitable as a multi-policy optimization technique for UTC [10] when deployed on noncooperating agents. Based on these results we hypothesize that W-Learning performance could be improved by enabling collaboration between agents.…”

Self-Adaptive Systems: A Systematic Literature Review Across Categories and Domains

“…Decentralized solutions were popular in automated traffic control [284], e-Commerce [264], water networks [88], service oriented systems [47], and load balancing solutions [16]. Learning was a prominent theme for automotive [89], traffic management control [90], and e-Commerce [132]. The ability for a system to adapt its adaptation logic was a strong area of focus for mobile systems [300], ISR [254], e-Commerce [9], software engineering applications [8], [106], and systems on chip [29].…”