HConfig: Resource Adaptive Fast Bulk Loading in HBase

Abstract

Abstract-NoSQL (Not only SQL) data stores become a vital component in many big data computing platforms due to its inherent horizontal scalability. HBase is an open-source distributed NoSQL store that is widely used by many Internet enterprises to handle their big data computing applications (e.g. Facebook handles millions of messages each day with HBase). Optimizations that can enhance the performance of HBase are of paramount interests for big data applications that use HBase or Big Table like key-value stores. In this paper we study the problems inherent in misconfiguration of HBase clusters, including scenarios where the HBase default configurations can lead to poor performance. We develop HConfig, a semiautomated configuration manager for optimizing HBase system performance from multiple dimensions. Due to the space constraint, this paper will focus on how to improve the performance of HBase data loader using HConfig. Through this case study we will highlight the importance of resource adaptive and workload aware auto-configuration management and the design principles of HConfig. Our experiments show that the HConfig enhanced bulk loading can significantly improve the performance of HBase bulk loading jobs compared to the HBase default configuration, and achieve 2~3.7x speedup in throughput under different client threads while maintaining linear horizontal scalability.

Revisiting Exact kNN Query Processing with Probabilistic Data Space Transformations

“…The schema used follows the aforementioned design; i.e., each row in this table has a rowkey that is the concatenation of a cluster ID and the lowerleft coordinates of a cell in the uniform space, and contains a single column with all data points (in the original space) mapped to that cell. Moreover, instead of inserting the data points into the table one by one, the standard HBase bulk loading [5] technique is used.…”

“…We also use the standard HBase bulk loading [5] technique to load the STOS-organised data into an HBase table. If the distribution of the row-keys of the HBase table is uniform, then the [5]-based loads are very efficient.…”

“…We also use the standard HBase bulk loading [5] technique to load the STOS-organised data into an HBase table. If the distribution of the row-keys of the HBase table is uniform, then the [5]-based loads are very efficient. Otherwise, a human expert is required to manually split the regions of the HBase table to expedite the bulk-loading process.…”

Policy-Driven Configuration Management for NoSQL

“…Due to the space constraint, we only include the results of read-intensive workloads. Also some detail on critical parameters and their impact on bulk loading performance of HBase are reported in [6]. Fig.4 shows the read throughput by varying the heap size from the 1GB default to 2GB, 4GB, 6GB.…”