A novel approach for efficient handling of small files in HDFS

Patel, Ankita; Mehta, Mayuri A.

doi:10.1109/iadcc.2015.7154903

Cited by 18 publications

(7 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their approach reduces memory consumption on the NameNode, but it didn't show how much the read and write performances are impacted. [16][17] [18] Y. Zhang et al proposed merging related small files according to WebGIS application, which improved the storage effeciency and HDFS metadata management, however the results are limited by the scene. D. Dev , al.…”

Section: Small File Problemmentioning

confidence: 99%

solution for the future: small file management by optimizing Hadoop

Achandair¹,

Bourekkadi²,

Elmahouti³

et al. 2018

IJET

View full text Add to dashboard Cite

Hadoop Distributed File System (HDFS) is designed to reliably store very large files across machines in a large cluster. It is one of the most used distributed file systems and offer a high availability and scalability on low-cost hardware. All Hadoopframework have HDFS as their storage component. Coupled with map reduce, which is the processing component, HDFS and Map Reduce (a processing component) have become the standard platforms for any management of big data in these days. HDFS however, in terms of design has the ability to handle huge numbers of large files, but when it comes to its deployments to handle large amounts of small files it might not be very effective. This paper puts forward a new strategy of managing small files. The approach will consists of two principal phases. The first phase will deal with the consolidating of aaclients input files, storing it continuously in a particular allocated block, that is a SequenceFile format, and so on into the next blocks. In this way we avoid the use of multiple block allocations for different streams, this reduces calls for available blocks and also reduces the metadata memory on the NameNode. Note the reason for this is that groups of small files packaged in a SequenceFile on the same block require one entry instead of one of each small file. The second phase will involve analyzing the attributes of stored small files so they can be distributed them in a way that the most called files will be referenced by an additional index as a MapFile format to reduce the read throughput during random access.

show abstract

Section: Small File Problemmentioning

confidence: 99%

solution for the future: small file management by optimizing Hadoop

Achandair¹,

Bourekkadi²,

Elmahouti³

et al. 2018

IJET

View full text Add to dashboard Cite

show abstract

“…In this respect, discovering knowledge out of huge volumes of data can be performed more efficiently, based on the high‐performance computing of cloud 21 . In addition, file correlations have become an increasingly important consideration for performance enhancement in various contexts, such as file systems, 22‐26 distributed systems, 27‐30 and cloud systems 3,18,31‐36 …”

Section: Introductionmentioning

confidence: 99%

Popularity and correlation aware data replication strategy based on half‐life concept and clustering in cloud system

Chellouf

Hamrouni

2020

Concurrency and Computation

View full text Add to dashboard Cite

Summary In the new millennium, a myriad of large‐scale applications (e.g., social networks, ecommerce, internet of things, and scientific experiments) often generate large volumes of data. Given their volumes, their heterogeneous and distributed nature, the management of such data constitutes a challenge for distributed systems, particularly cloud computing. In this regard, data replication is a well‐known and effective data management technique that consists in creating multiple copies of the same data in different storage resources. In this article, we propose a popularity and correlation based data replication strategy, called PCDR. The main idea of our strategy is to replicate a set of the most popular correlated file groups based on files access history analysis. In this respect, the popularity of the files in each group is determined while taking into consideration temporal locality. Moreover, a clustering technique is used to find groups of correlated files. Using the CloudSim simulator, extensive experimentations show that our proposed strategy outperforms other strategies for several evaluation metrics.

show abstract

“…In order to meet the characteristics of WebGIS accessing patterns, Liu et al [ 8 ] proposed WebGIS-based HDFS storage to support popular web applications, but their method is sensitive to small file I/O performance, and the accessing time of their method is still relatively long. Being aware of the quality of service of the DataNodes, Cloud storage from HDFS [ 9 , 10 ] can increase the storage utilization ratio and improve storage performance. This makes the Hadoop distributed computing model more suitable to unstable wide area network environments.…”

Section: Introductionmentioning

confidence: 99%

Efficient Retrieval of Massive Ocean Remote Sensing Images via a Cloud-Based Mean-Shift Algorithm

Yang

Song

Mei

2017

Sensors

View full text Add to dashboard Cite

The rapid development of remote sensing (RS) technology has resulted in the proliferation of high-resolution images. There are challenges involved in not only storing large volumes of RS images but also in rapidly retrieving the images for ocean disaster analysis such as for storm surges and typhoon warnings. In this paper, we present an efficient retrieval of massive ocean RS images via a Cloud-based mean-shift algorithm. Distributed construction method via the pyramid model is proposed based on the maximum hierarchical layer algorithm and used to realize efficient storage structure of RS images on the Cloud platform. We achieve high-performance processing of massive RS images in the Hadoop system. Based on the pyramid Hadoop distributed file system (HDFS) storage method, an improved mean-shift algorithm for RS image retrieval is presented by fusion with the canopy algorithm via Hadoop MapReduce programming. The results show that the new method can achieve better performance for data storage than HDFS alone and WebGIS-based HDFS. Speedup and scaleup are very close to linear changes with an increase of RS images, which proves that image retrieval using our method is efficient.

show abstract

A novel approach for efficient handling of small files in HDFS

Cited by 18 publications

References 9 publications

solution for the future: small file management by optimizing Hadoop

solution for the future: small file management by optimizing Hadoop

Popularity and correlation aware data replication strategy based on half‐life concept and clustering in cloud system

Efficient Retrieval of Massive Ocean Remote Sensing Images via a Cloud-Based Mean-Shift Algorithm

Contact Info

Product

Resources

About