Multi-objective Reinforcement Learning for Responsive Grids

Abstract

Grids organize resource sharing, a fundamental requirement of large scientific collaborations. Seamless integration of Grids into everyday use requires responsiveness, which can be provided by elastic Clouds, in the Infrastructure as a Service (IaaS) paradigm. This paper proposes a model-free resource provisioning strategy supporting both requirements. Provisioning is modeled as a continuous action-state space, multiobjective reinforcement learning (RL) problem, under realistic hypotheses; simple utility functions capture the high level goals of users, administrators, and shareholders. The model-free approach falls under the general program of autonomic computing, where the incremental learning of the value function associated with the RL model provides the so-called feedback loop. The RL model includes an approximation of the value function through an Echo State Network. Experimental validation on a real data-set from the EGEE Grid shows that introducing a moderate level of elasticity is critical to ensure a high level of user satisfaction.

Softmax exploration strategies for multiobjective reinforcement learning

“…Otherwise an action is selected randomly. This has been the predominant exploration approach adopted in the MORL literature so far [12,15,16,19,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45].…”

Cost-Wait Trade-Offs in Client-Side Resource Provisioning with Elastic Clouds

“…Even though the aim is to improve globally the satisfaction, they employ techniques which could be transposed to a client-side approach. Perez et al [11] propose a learning scheme to prioritize the scheduling in presence of mixed workloads (interactive and best-effort jobs), with a bi-objective optimization problem: fairness among users and responsiveness of job requests. A similar goal is targeted by Quiroz et al in [12], which propose an online clustering approach to detect patterns in the stream of requests.…”

A comparative study between motivated learning and reinforcement learning

“…where (2) represents the availability of resource Si' Rc is a current resource amount, and Rd is a desired resource amount established by the ML agent during learning. (The implementation used in this work simply sets Rd equal to the initial environment state.)…”

“…Existing approaches that aim at multi-objective reinforcement learning [2], [3] assume stationary operating conditions. Existing task coordination strategies include resource sharing [4], learning of coordination [5] and the use of set strategies [6].…”