HBA: Distributed Metadata Management for Large Cluster-Based Storage Systems

Zhu, Yan; Jiang, Hong; Wang, Jim; Xian, Feng

doi:10.1109/tpds.2007.70788

Cited by 64 publications

(5 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the method does not consider the case of increasing or deleting servers. Hua et al (2008) and Zhu et al (2008) used the bloom filter technology to efficiently locate MDSs without considering metadata allocation or load balancing issues. The hashing method distributes metadata among servers based on hash values.…”

Section: Related Workmentioning

confidence: 99%

MDLB: a metadata dynamic load balancing mechanism based on reinforcement learning

Jin

Zhang

et al. 2020

Front Inform Technol Electron Eng

View full text Add to dashboard Cite

With the growing amount of information and data, object-oriented storage systems have been widely used in many applications, including the Google File System, Amazon S3, Hadoop Distributed File System, and Ceph, in which load balancing of metadata plays an important role in improving the input/output performance of the entire system. Unbalanced load on the metadata server leads to a serious bottleneck problem for system performance. However, most existing metadata load balancing strategies, which are based on subtree segmentation or hashing, lack good dynamics and adaptability. In this study, we propose a metadata dynamic load balancing (MDLB) mechanism based on reinforcement learning (RL). We learn that the Q_learning algorithm and our RL-based strategy consist of three modules, i.e., the policy selection network, load balancing network, and parameter update network. Experimental results show that the proposed MDLB algorithm can adjust the load dynamically according to the performance of the metadata servers, and that it has good adaptability in the case of sudden change of data volume.

show abstract

Section: Related Workmentioning

confidence: 99%

MDLB: a metadata dynamic load balancing mechanism based on reinforcement learning

Jin

Zhang

et al. 2020

Front Inform Technol Electron Eng

View full text Add to dashboard Cite

show abstract

“…For instance, transmitting compressed indicators, at the expense of larger local memory consumption and computational work in the transmitting node [18]. Another approach is to accurately advertise important information while allowing less critical data to be stale, or less accurate [19], [20]. The work [14] surveys many optimizations to indicators, such as the support for removals and dynamic scaling.…”

Section: Related Work a Cache-indicators Systemsmentioning

confidence: 99%

Self-adjusting Advertisement of Cache Indicators with Bandwidth Constraints

Cohen

Einziger

Scalosub

2021

IEEE INFOCOM 2021 - IEEE Conference on Computer Communications

View full text Add to dashboard Cite

Caching is extensively used in various networking environments to optimize performance by reducing latency, bandwidth, and energy consumption. To optimize performance, caches often advertise their content using indicators, which are data structures that trade space efficiency for accuracy. However, this tradeoff introduces the risk of false indications. Existing solutions for cache content advertisement and cache selection often lead to inefficiencies, failing to adapt to dynamic network conditions. This paper introduces SALSA2, a Scalable Adaptive and Learning-based Selection and Advertisement Algorithm, which addresses these limitations through a dynamic and adaptive approach. SALSA2 accurately estimates mis-indication probabilities by considering inter-cache dependencies and dynamically adjusts the size and frequency of indicator advertisements to minimize transmission overhead while maintaining high accuracy. Our extensive simulation study, conducted using a variety of realworld cache traces, demonstrates that SALSA2 achieves up to 84% bandwidth savings compared to the state-of-the-art solution and close-to-optimal service cost in most scenarios. These results highlight SALSA2's effectiveness in enhancing cache management, making it a robust and versatile solution for modern networking challenges.

show abstract

“…The key issue is the timing of the accumulation update, take the write OSD operation into account to merge the cumulative update and simplification [9,16] is an effective way. Bloom filter matrix storing metadata hash values and unconsolidated update sequence is adopted before Simplification and update operations.…”

Section: Figure 2 the Structure Of Cache Poolmentioning

confidence: 99%

“…As we know, two types of the main server is responsible for service for the object storage system, one is responsible for providing metadata, namely MDS, the other is responsible for providing storage information, namely OSD [9][10][11][12]. In order to improve the difference of efficiency and the performance between the two types of servers and services, a two level structure is adopted, in which the MDS cache in the first-level is called L1 cache, while the OSD cache in the second-level is called L2 cache.…”

Section: Introductionmentioning

confidence: 99%

A Cache Design of Load Balancing System for Object-Based Storage

Shan¹,

Liu²,

Yao

2015

IJMUE

View full text Add to dashboard Cite

In design of the object-based storage system, cache design of the server becomes an important guarantee for improving service quality. This paper presents a cache design of load balancing system for object-based storage, which consists of three main aspects. First, the cache update policy is proposed. The strategy considers the overall cache as a pool, consistency and accuracy of updating is ensured through the locking mechanism, while using Bloom Filter combined with the updated time series to achieve updating simplification. Secondly, the cache replacement strategy is introduced which achieving the cache replacement policy using cache spanning tree combined with cache stack. Finally, cache design model is proposed based on energy consumption. Experimental results show that the proposed cache design can improve the efficiency of cache operation in object-based storage system, while reducing energy consumption of cache operation. 132Copyright ⓒ 2015 SERSC energy consumption cache design is adopted which ensure consistency, isolation, durability and stability of cache operations to the greatest extent to obtain the best performance. Related WorksIn recent years, there are many research directions on cache design, in which the research for the study of energy consumption and cache collaborative aspects of services has become a hot research and focus.[13] The two-level structure of the disk cache management and the corresponding management algorithms is proposed to ensure the overall performance of the system under high load conditions. [14] The client and metadata server cache is designed as a whole, while the cooperative caching scheme based on the object's size, access costs and network load, can effectively improve the input-output performance of the system. [15] proposed energy-related storage cache management, offline caching algorithms by proposing energy-related greedy algorithms, online energy-related algorithms, the energy consumption of the disk.[16] proposed a local algorithm and a multi-queue management algorithm with global multi-level buffer cache hierarchy, thereby increasing the actual hit rate of second-level buffer cache. Hierarchical bloom filter arrays (HBA): A novel, scalable metadata management system for large cluster-based storage

show abstract

HBA: Distributed Metadata Management for Large Cluster-Based Storage Systems

Cited by 64 publications

References 36 publications

MDLB: a metadata dynamic load balancing mechanism based on reinforcement learning

MDLB: a metadata dynamic load balancing mechanism based on reinforcement learning

Self-adjusting Advertisement of Cache Indicators with Bandwidth Constraints

A Cache Design of Load Balancing System for Object-Based Storage

Contact Info

Product

Resources

About