Study on Replica Strategy of Big Data Storage based on Cloud Environment

Jun, Qin; Liang, Shengxi; Song, Yanyan; Zong, Ping

doi:10.1109/iccse49874.2020.9201815

Cited by 3 publications

(1 citation statement)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The authors in [ 36 ] proposed a replica placement model based on evaluating the comprehensive performance value of the node. The proposed model used the Hadoop Distributed File System (HDFS) as the base model to be improved which divide the data into small blocks as the basic storage unit to be stored on different distributed nodes.…”

Section: Related Workmentioning

confidence: 99%

Replicating File Segments between Multi-Cloud Nodes in a Smart City: A Machine Learning Approach

Mostafa

Kotb

Al-Arnaout

et al. 2023

Sensors

View full text Add to dashboard Cite

The design and management of smart cities and the IoT is a multidimensional problem. One of those dimensions is cloud and edge computing management. Due to the complexity of the problem, resource sharing is one of the vital and major components that when enhanced, the performance of the whole system is enhanced. Research in data access and storage in multi-clouds and edge servers can broadly be classified to data centers and computational centers. The main aim of data centers is to provide services for accessing, sharing and modifying large databases. On the other hand, the aim of computational centers is to provide services for sharing resources. Present and future distributed applications need to deal with very large multi-petabyte datasets and increasing numbers of associated users and resources. The emergence of IoT-based, multi-cloud systems as a potential solution for large computational and data management problems has initiated significant research activity in the area. Due to the considerable increase in data production and data sharing within scientific communities, the need for improvements in data access and data availability cannot be overlooked. It can be argued that the current approaches of large dataset management do not solve all problems associated with big data and large datasets. The heterogeneity and veracity of big data require careful management. One of the issues for managing big data in a multi-cloud system is the scalability and expendability of the system under consideration. Data replication ensures server load balancing, data availability and improved data access time. The proposed model minimises the cost of data services through minimising a cost function that takes storage cost, host access cost and communication cost into consideration. The relative weights between different components is learned through history and it is different from a cloud to another. The model ensures that data are replicated in a way that increases availability while at the same time decreasing the overall cost of data storage and access time. Using the proposed model avoids the overheads of the traditional full replication techniques. The proposed model is mathematically proven to be sound and valid.

show abstract

Section: Related Workmentioning

confidence: 99%

Replicating File Segments between Multi-Cloud Nodes in a Smart City: A Machine Learning Approach

Mostafa

Kotb

Al-Arnaout

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

DRF-FTS: A Dynamic Replication Factor Replication Scheme Based on Fault-Tolerant Set

Yan

Song

Wang

2021

2021 IEEE 3rd International Conference on Civil Aviation Safety and Information Technology (ICCASIT)

View full text Add to dashboard Cite

Intelligent Replica Selection in Edge and IoT Environments Using Artificial Neural Networks

et al. 2022

View full text Add to dashboard Cite

Cloud, edge and Internet of Things (IoT) technologies have emerged to overcome the challenges involved in sharing computational resources and information services. Within generic cloud systems, two models have been identified as having widespread applicability: computation clouds and data clouds. A data cloud is cloud computing that aims to manage, unify and operate multiple data workloads. Many current applications generate datasets consisting of petabytes (PB) of information. Managing large datasets is a complex issuel; in particular, datasets associated with many applications can be distributed widely in geographical terms, particularly in IoT systems. Edge and IoT systems are facing new challenges with increased complexity, making scalability an important issue that will affect the performance of the system. Data replication services are widely accepted techniques to improve availability and fault tolerance, and to improve the data access time. Current replication services, however, often exhibit an increase in response time, reflecting the problems associated with the ever-increasing size of databases. This paper proposes a prediction model to predict replica locations using the files’ access profile, which feeds the neural networks with the access and location behavior (file profile) to minimize the overhead of transferring large volumes of data, which slows down the system and requires careful management. This new model has shown high accuracy and low overheads. The result shows a significant improvement in total task execution time using the proposed model for locating files by 16.34% and 30.45%; in addition, the results show bandwidth improvement by 24.7% and 49.4% compared to the user profile prediction model and replica service model without prediction, respectively. Consequently, the proposed algorithm can improve data access speed, reduce data access latency and decrease bandwidth consumption.

show abstract

Study on Replica Strategy of Big Data Storage based on Cloud Environment

Cited by 3 publications

References 3 publications

Replicating File Segments between Multi-Cloud Nodes in a Smart City: A Machine Learning Approach

Replicating File Segments between Multi-Cloud Nodes in a Smart City: A Machine Learning Approach

DRF-FTS: A Dynamic Replication Factor Replication Scheme Based on Fault-Tolerant Set

Intelligent Replica Selection in Edge and IoT Environments Using Artificial Neural Networks

Contact Info

Product

Resources

About