Data Replication and Placement Strategies in Distributed Systems: A State of the Art Survey

Sabaghian, Khatereh; Khamforoosh, Keyhan; Ghaderzadeh, Abdulbaghi

doi:10.1007/s11277-023-10240-7

Cited by 2 publications

(2 citation statements)

References 99 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Modern applications demand distributed storage solutions that are not only fault-tolerant and storage-efficient but also efficient in terms of performance and data integrity simultaneously. Researchers have proposed various techniques to deal with data reliability [15]. To ensure data reliability, the replication strategy is the common redundancy mechanism used for distributed storage systems.…”

Section: Related Workmentioning

confidence: 99%

Robust-DSN: A Hybrid Distributed Replication and Encoding Network Grouped with a Distributed Swarm Workflow Scheduler

Hameed,

Barzegar,

Ioini

et al. 2024

Electronics

View full text Add to dashboard Cite

In many distributed applications such as the Internet of Things (IoT), large amounts of data are being generated that require robust storage solutions. Traditional cloud solutions, although efficient, often lack trust and transparency because of centralized management. To address these issues, we present Robust-DSN, a distributed storage network leveraging the hybrid distributed replication and encoding network (HYDREN) and the distributed swarm workflow scheduler (DSWS) as its main components. Our system uses an interplanetary file system (IPFS) as an underlay storage network and segments it into multiple regions to distribute the failure domain and improve the data’s proximity to users. HYDREN incorporates Reed–Solomon encoding and distributed replication to improve file availability, while DSWS optimizes resource allocation across the network. The uploaded file is encoded into chunks and distributed across distinct optimal nodes leveraging lightweight multithreading. Additionally, Robust-DSN verifies the integrity of all chunks by preserving the hashes when uploading and validating each chunk while downloading. The proposed system provides a comprehensive solution for resilient distributed data storage, focusing on the key challenges of data availability, integrity, and performance. The results reveal that compared with a state-of-the-art system, the proposed system improves file recovery by 15%, even with a 50% peer failure rate. Furthermore, with replication factor 4 and the same failure resilience as IPFS, it saves 50% storage and enhances file recovery by 8%. Robust-DSN acts as a distributed storage platform for emerging technologies, expanding storage system capabilities in a wide range of distributed applications.

show abstract

Section: Related Workmentioning

confidence: 99%

Robust-DSN: A Hybrid Distributed Replication and Encoding Network Grouped with a Distributed Swarm Workflow Scheduler

Hameed,

Barzegar,

Ioini

et al. 2024

Electronics

View full text Add to dashboard Cite

show abstract

“…This results in an unequal number of data blocks in each VM, regardless of its performance and location. A detailed description of different replication strategies in distributed file systems on the cloud is mentioned in [17,18].…”

Section: Related Workmentioning

confidence: 99%

A Novel Data Management Scheme in Cloud for Micromachines

Singh,

Jeyaraj,

Sharma

et al. 2023

Electronics

View full text Add to dashboard Cite

In cyber-physical systems (CPS), micromachines are typically deployed across a wide range of applications, including smart industry, smart healthcare, and smart cities. Providing on-premises resources for the storage and processing of huge data collected by such CPS applications is crucial. The cloud provides scalable storage and computation resources, typically through a cluster of virtual machines (VMs) with big data tools such as Hadoop MapReduce. In such a distributed environment, job latency and makespan are highly affected by excessive non-local executions due to various heterogeneities (hardware, VM, performance, and workload level). Existing approaches handle one or more of these heterogeneities; however, they do not account for the varying performance of storage disks. In this paper, we propose a prediction-based method for placing data blocks in virtual clusters to minimize the number of non-local executions. This is accomplished by applying a linear regression algorithm to determine the performance of disk storage on each physical machine hosting a virtual cluster. This allows us to place data blocks and execute map tasks where the data blocks are located. Furthermore, map tasks are scheduled based on VM performance to reduce job latency and makespan. We simulated our ideas and compared them with the existing schedulers in the Hadoop framework. The results show that the proposed method improves MapReduce performance in terms of job latency and makespan by minimizing non-local executions compared to other methods taken for evaluation.

show abstract

Data Replication and Placement Strategies in Distributed Systems: A State of the Art Survey

Cited by 2 publications

References 99 publications

Robust-DSN: A Hybrid Distributed Replication and Encoding Network Grouped with a Distributed Swarm Workflow Scheduler

Robust-DSN: A Hybrid Distributed Replication and Encoding Network Grouped with a Distributed Swarm Workflow Scheduler

A Novel Data Management Scheme in Cloud for Micromachines

Contact Info

Product

Resources

About