Data Replication in Data Intensive Scientific Applications with Performance Guarantee

Nukarapu, D. T.; Tang, Bin; Wang, Liqiang; Lu, Shiyong

doi:10.1109/tpds.2010.207

Cited by 72 publications

(33 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The most directly related work to this replication work is complicated process on clouds server. The data replication and request response on cloud server as a static optimization problem on user access [7]. They show that this problem is NP-hard and request delay, which means that present, is no polynomial algorithm that provides an accurate solution.…”

Section: Related Workmentioning

confidence: 99%

Popularity (Hit Rate) Based Replica Creation for Enhancing the Availability in Cloud Storage

Selvi¹,

Anbuselvi²

2018

IJIES

View full text Add to dashboard Cite

Abstract:In cloud computing, the replication management system has been well adopted in cloud storage applications. To provide the availability and reliability, the replication system replicates the files and can be stored in different server. The system led some complicated issues such as high memory consumption, incurred high storage cost and to access the file is more complicated issues in recent cloud storage applications. In the existing technique, File Accessing Frequency based Ranking (FAFR) Algorithm and Dynamically Reduced Replica for Rarely Accessed files (DRRRA) algorithm work jointly and identify the rarely accessed files and retain the replica in two server other replicated files are deleted. To provide access to more request with 2 or 3-replica is a complicated issue. Thus, this paper proposes a Dynamic replica Creation for Availability enhanced Storage (DRCAES) algorithm which jointly work with FAFR algorithm to predict most frequently accessed files and automatically replicated to other server based on server memory. The aim of this proposed approach is to enhance the availability, thereby reducing the request-response delay time. Thus the proposed approach optimizes the number of replicas, occupied space, and cost.

show abstract

Section: Related Workmentioning

confidence: 99%

Popularity (Hit Rate) Based Replica Creation for Enhancing the Availability in Cloud Storage

Selvi¹,

Anbuselvi²

2018

IJIES

View full text Add to dashboard Cite

show abstract

“…The design of NFS involves simplicity and hence they did not take into consideration any of the complex concurrent read/write issues. Dharma et al [38] propose a data replication algorithm that not only has a provable theoretical performance guarantee, but also can be implemented in a distributed and practical manner. Specifically, authors have designed a polynomial time centralized replication algorithm that reduces the total data file access delay by at least half of that reduced by the optimal replication solution.…”

Section: Related Workmentioning

confidence: 99%

A dynamic file replication based on CPU load and consistency mechanism in a trusted distributed environment

Vardhan

Kushwaha

2017

Teh. vjesn.

View full text Add to dashboard Cite

Original scientific paper An effort has been made to propose a CPU load based dynamic, cooperative, trust based, and secure file replication approach based along with consistency among file replicas for distributed environment. Simulation results consisting of 100 requesting nodes, three file servers and file size ranging from 677 KB to 11 MB establishes that, when the CPU load is taken into consideration, the average decrease in file request completion time is about 22,04 ÷ 24,81 % thus optimizing the CPU load and minimizing the file request completion time. The CPU load decreases by 4,25 ÷ 5,58 %. Results show that, the average write latency with proposed mechanism decreases by 6,12 % as compared to Spinnaker writes and the average read latency is 3 times better than Cassandra Quorum Read (CQR). The proposed partial update propagation for maintaining file consistency stands to gain up to 69,67 % in terms of time required to update stale replicas. Thus the integrity of files and behaviour of the requesting nodes and file servers is guaranteed within even lesser time. Finally, a relationship between the formal aspects of simple security model and secure reliable CPU load based file replication model is established through process algebra. Keywords Dinamička replikacija datoteke zasnovana na mehanizmu opterećenosti i konzistencije CPU u pouzdanom distribuiranom okruženjuIzvorni znanstveni članak Pokušalo se predložiti dinamički, kooperativni, pouzdani i sigurni pristup replikaciji datoteke utemeljen na opterećenosti CPU uz konzistenciju među replikama datoteke za distribuirano okruženje. Rezultati simulacije koja se sastoji od 100 potrebnih čvorova, tri servera datoteke i datoteke veličine od 677 KB to 11 MB pokazuju da kada se uzme u obzir opterećenje CPU, prosječno smanjenje vremena potrebnog za popunjavanje datoteke je oko 22,04 ÷ 24,81 %. Tako se optimiziralo opterećenje CPU i smanjilo traženo vrijeme popunjavanja datoteke. Opterećenje CPU smanjuje se za 4,25 ÷ 5,58 %. Rezultati pokazuju da se prosječno kašnjenje upisa (write latency) s predloženim mehanizmom smanjuje za 6,12 % u usporedbi sa Spinnakerovim, a prosječno vrijeme čekanja čitanja (read latency) je 3 puta bolje od Cassandra Quorum Read (CQR). Predložena parcijalna propagacija ažuriranja za održavanje konzistencije datoteke povećava se do 69,67 % u odnosu na vrijeme potrebno za ažuriranje zastarjelih replika. Tako je integritet datoteka i ponašanje zahtijevanih čvorova i servera datoteke zagarantirano za čak manje vremena. Konačno, kroz algebra postupak uspostavljen je odnos između formalnih aspekata jednostavnog modela sigurnosti i sigurnog pouzdanog modela replikacije datoteke zasnovanog na sigurnom pouzdanom opterećenju datoteke.

show abstract

“…It's based on a masterslave architecture, the NameNode as a master and the DataNodes as slaves. The NameNode is responsible for managing the file system namespace, it keeps tracks of files during creation, deletion, replication [3] and manages all the related metadata [4] in the server memory. The NameNode splits files into blocks and sends the writes requests to be performed locally by DataNodes.…”

Section: Hadoop Storagementioning

confidence: 99%

Optimizing Hadoop for Small File Management

Elmahouti

Achandair

Khoulji

et al. 2017

TMLAI

View full text Add to dashboard Cite

HDFS is one of the most used distributed file systems, that offer a high availability and scalability on low cost hardware. HDFS is delivered as the storage component of Hadoop framework. Coupled with map reduce, which is the processing component, HDFS and MapReduce become the de facto platform for managing big data nowadays. However, HDFS was designed to handle specifically a huge number of large files, while when it comes to a large number of small files, Hadoop deployments may be not efficient. In this paper, we proposed a new strategy to manage small files. Our approach consists of two principal phases. The first phase is about consolidating more than only one client's small files input, and store the inputs continuously in the first allocated block, in a SequenceFile format, and so on into the next blocks. That way we avoid multiple block allocations for different streams, to reduce calls for available blocks and to reduce the metadata memory on the NameNode. This is because groups of small files packaged in a SequenceFile on the same block will require one entry instead of one for each small file. The second phase consists of analyzing attributes of stored small files to distribute them in such a way that the most called files will be referenced by an additional index as a MapFile format to reduce the read throughput during random access.

show abstract

Data Replication in Data Intensive Scientific Applications with Performance Guarantee

Cited by 72 publications

References 44 publications

Popularity (Hit Rate) Based Replica Creation for Enhancing the Availability in Cloud Storage

Popularity (Hit Rate) Based Replica Creation for Enhancing the Availability in Cloud Storage

A dynamic file replication based on CPU load and consistency mechanism in a trusted distributed environment

Optimizing Hadoop for Small File Management

Contact Info

Product

Resources

About