Hadoop high availability through metadata replication

Wang, Feng; Qiu, Jie; Yang, Jie; Dong, Bo; Li, Xinhui; Liu, Ying

doi:10.1145/1651263.1651271

Cited by 100 publications

(49 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many researchers have been done using failure detection and prediction so that Hadoop can overcome issues with NameNode. Feng Wang and his colleges [7] used metadata replication scheme to enhance Hadoop NameNode High Availability. A different approach is presented by Cristina L. Abad and his buddies [2].…”

Section: Related Workmentioning

confidence: 99%

Enhancing NameNode Fault Tolerance in Hadoop Distributed File System

Aung¹,

Thein²

2014

IJCA

View full text Add to dashboard Cite

In today's cloud computing environment, Hadoop is applied for handling huge data, tens of terabytes to petabytes, with commodity hardware (HDFS) for storage and software (MapReduce) for parallel data processing. In Hadoop version 1.0.3, there is a single metadata server called NameNode which stores the entire file system metadata in main memory and most of I/O operations are associated with those credential metadata. Hadoop is out of commission if NameNode is crashed because it works on memory which becomes exhausted due to multiple concurrent accesses [3]. Therefore, NameNode is a single point of failure (SPOF) in Hadoop and it has to tolerate faults. To solve this issue, a proactive predictive solution is proposed for enhancing NameNode fault tolerance. The solution is designed to proactively calculate the predicted time to crash of NameNode due to resource exhaustion by evaluating the use of powerful Back Propagation Algorithm Neural Network. The proposed approach can give prediction accuracy with minimal error compared to the actual result. Therefore, NameNode's single point of failure can overcome through proposed proactively predicting the time to crash of NameNode caused by memory resource exhaustion.

show abstract

Section: Related Workmentioning

confidence: 99%

Enhancing NameNode Fault Tolerance in Hadoop Distributed File System

Aung¹,

Thein²

2014

IJCA

View full text Add to dashboard Cite

show abstract

“…Feng Wang's discussed the metadata replication based solution to provide high availability and failover technique [9]. The solution has phases: the first is the initialization phase which initializes the execution environment of high availability.…”

Section: Background and Related Workmentioning

confidence: 99%

High Scalability of HDFS using Distributed Namespace

Singh¹,

Singh²

2012

IJCA

View full text Add to dashboard Cite

In data intensive computing, Hadoop is widely used by organizations. The client applications of Hadoop require high availability and scalability of the system. Mostly, these applications are online and their data growth rate is unpredictable. The present Hadoop relies on secondary namenode for failover which slows down the performance of the system. Hadoop system's scalability depends on the vertical scalability of namenode server. As the namespace of Hadoop distributed file system grows, it demands additional memory to cache. A namenode server does not have enough primary memory to cache the namespace, its performance and availability effects. A new Hadoop architecture has been proposed to address the issues of namenode scalability, single point of failure and availability of Hadoop. This approach is based on distribution of namespace using distributed hash tables. The growing size of namespace of HDFS is distributed into multiple name node servers. The proposed architecture of Hadoop is simulated by using the multiple name node servers. The name node are arranges in chord ring. This allows HDFS to scale up horizontally. The system provides decartelize managed approach for namespace distribution which gives consistent performance. The results of HDFS namespace to store 1 billion or above files are discussed in this research work. The proposed architecture has shown high availability and adapts to name node failure. General TermsData intensive computing, Scalability, Failover, Availability

show abstract

“…Data replication technique is widely used in the cloud computing system to provide high data availability [13], [14], [15], [16]. While research has been done on replica placement, little has focused on the dynamic maintenance of the replicas over the time.…”

Section: Introductionmentioning

confidence: 99%

On the Dynamic Maintenance of Data Replicas based on Access Patterns in A Multi-Cloud Environment

Shorfuzzaman

2017

ijacsa

View full text Add to dashboard Cite

Abstract-Cloudcomputing provides services and infrastructures to enable end-users to access, modify and share massive geographically distributed data. There are increasing interests in developing data-intensive (big data) applications in this computing environment that need to access huge datasets. Accessing such data in an efficient way is deterred with factors such as dynamic changes in resource availability, provision of diverse service quality by different cloud providers. Data replication has already been proven to be an effective technique to overcome these challenges. Replication offers reduced response time in data access, higher data availability and improved system load balancing. Once the replicas are created in a multi-cloud environment, it is of utmost importance to continuously support maintenance of these replicas dynamically. This is to ensure that replicas are located in optimal data center locations to minimize replication cost and to meet specific user and system requirements. First, this paper proposes a novel approach to distributed placement of static replicas in appropriate data center locations. Secondly, motivated by the fact that a multi-cloud environment is highly dynamic, the paper presents a dynamic replica maintenance technique that reallocates replicas to new data center locations upon significant performance degradation. The evaluation results demonstrate the effectiveness of the presented dynamic maintenance technique with static placement decisions in a multi-cloud environment.

show abstract

Hadoop high availability through metadata replication

Cited by 100 publications

References 3 publications

Enhancing NameNode Fault Tolerance in Hadoop Distributed File System

Enhancing NameNode Fault Tolerance in Hadoop Distributed File System

High Scalability of HDFS using Distributed Namespace

On the Dynamic Maintenance of Data Replicas based on Access Patterns in A Multi-Cloud Environment

Contact Info

Product

Resources

About