MDLB: a metadata dynamic load balancing mechanism based on reinforcement learning

Abstract

With the growing amount of information and data, object-oriented storage systems have been widely used in many applications, including the Google File System, Amazon S3, Hadoop Distributed File System, and Ceph, in which load balancing of metadata plays an important role in improving the input/output performance of the entire system. Unbalanced load on the metadata server leads to a serious bottleneck problem for system performance. However, most existing metadata load balancing strategies, which are based on subtree segmentation or hashing, lack good dynamics and adaptability. In this study, we propose a metadata dynamic load balancing (MDLB) mechanism based on reinforcement learning (RL). We learn that the Q_learning algorithm and our RL-based strategy consist of three modules, i.e., the policy selection network, load balancing network, and parameter update network. Experimental results show that the proposed MDLB algorithm can adjust the load dynamically according to the performance of the metadata servers, and that it has good adaptability in the case of sudden change of data volume.

A Blind Load-Balancing Algorithm (BLBA) for Distributing Tasks in Fog Nodes

“…The objective of this study is to optimize load-balancing and migration cost while satisfying the delay constraints of the computation task of vehicles. Wu et al [27] proposed a reinforcement learning-based metadata dynamic load-balancing mechanism. This method can control the load dynamically according to the performance of the metadata servers, and it has good adaptability in the case of a sudden change in data volume.…”

“…To our knowledge, most of the methods presented in previous research for decisionmaking require knowledge of the capacity of neighbor FNs and the cloud (e.g., [10,26,27]), which creates a traffic load on the network and also delays the decision-making process. Our work in this paper differs from previous works as in our method, the FN just decides based on the information obtained during the learning period from delay and reward based on its own condition and has no knowledge of the status of neighbor nodes.…”