RACAM: design and implementation of a recursively adjusting co‐allocation method with efficient replica selection in Data Grids

Abstract

SUMMARYData Grids enable the sharing, selection, and connection of a wide variety of geographically distributed computational and storage resources for addressing large-scale data-intensive scientific application needs in, for instance, high-energy physics, bioinformatics, and virtual astrophysical observatories. Data sets are replicated in Data Grids and distributed among multiple sites. Unfortunately, data sets of interest sometimes are significantly large in size, and may cause access efficiency overhead. A co-allocation architecture was developed in order to enable parallel downloading of data sets from multiple servers. Several co-allocation strategies have been coupled and used to exploit download rate by specifying among various client-server divides files into multiple blocks of equal sizes to link and address dynamic rate fluctuations. However, one major obstacle, the idle time of faster servers having to wait for the slowest server to deliver the final block, makes it important to reduce differences in finishing time among replica servers. In this paper, we propose a dynamic co-allocation method, called Recursively Adjusting Co-Allocation Method (RACAM), to improve the performance of parallel data file transfer. Our approach reduces the idle time spent waiting for the slowest server and decreases data transfer completion time. We also provide an effective scheme for reducing the cost of reassembling data blocks.

Accessing medical image file with co-allocation HDFS in cloud

“…Previous research [11,13,22,24,25,27] shows that co-allocation can solve the data transfer problem in grid environments; these results are categorized: RAM [12,24,22], DAS, [27,13] and ARAM [28,23]. This concept is the foundation for the current study.…”

“…Building cloud computing in this large-scale parallel computing cluster is growing with thousands of processors [7,8,3,6,[9][10][11][12][13][14][15][16][17][18][19][20][21]. In such a large number of compute nodes, faults are becoming common place.…”