Classification Based Metadata Management for HDFS

For scalable data storage, Hadoop is widely used nowadays. It provides a distributed file system that stores data on the compute nodes. Basically, it represents a master/slave architecture that consists of a NameNode and copious Data Nodes. Data Nodes contain application data and metadata of application data resides in the Main Memory of NameNode. In cached approach, they fragment the metadata depending on the last access time and move the least frequently used data to secondary memory. If the requested data is not found in main memory then the secondary data will be loaded again on the RAM. So when the secondary data reloads to the primary memory then the NameNode main memory limitation arises again. The focus of this research is to reduce the namespace problem of main memory and to make the system dynamically scalable. A new Metadata Fragmentation Algorithm is proposed that separates the metadata list of NameNode dynamically. The NameNode creates Secondary Memory File in perspective of the threshold value and allocates secondary memory location based on the requirement. According to the proposed algorithm the maximum third, out of fourth of main memory is used at the secondary file caching time. The free space aids in faster operation by Dynamically Scalable NameNode approach. This proposed algorithm shows that the space utilization is increased to 17% and time utilization is increased to 0.0005% with the comparison of the existing fragmentation algorithm.For scalable data storage, Hadoop is widely used nowadays. It provides a distributed file system that stores data on the compute nodes. Basically, it represents a master/slave architecture that consists of a NameNode and copious Data Nodes. Data Nodes contain application data and metadata of application data resides in the Main Memory of NameNode. In cached approach, they fragment the metadata depending on the last access time and move the least frequently used data to secondary memory. If the requested data is not found in main memory then the secondary data will be loaded again on the RAM. So when the secondary data reloads to the primary memory then the NameNode main memory limitation arises again. The focus of this research is to reduce the namespace problem of main memory and to make the system dynamically scalable. A new Metadata Fragmentation Algorithm is proposed that separates the metadata list of NameNode dynamically. The NameNode creates Secondary Memory File in perspective of the threshold value and allocates secondary memory location based on the requirement. According to the proposed algorithm the maximum third, out of fourth of main memory is used at the secondary file caching time. The free space aids in faster operation by Dynamically Scalable NameNode approach. This proposed algorithm shows that the space utilization is increased to 17% and time utilization is increased to 0.0005% with the comparison of the existing fragmentation algorithm.For scalable data storage, Hadoop is widely used nowadays. It provides a distributed file system that stores data on the compute nodes. Basically, it represents a master/slave architecture that consists of a NameNode and copious Data Nodes. Data Nodes contain application data and metadata of application data resides in the Main Memory of NameNode. In cached approach, they fragment the metadata depending on the last access time and move the least frequently used data to secondary memory. If the requested data is not found in main memory then the secondary data will be loaded again on the RAM. So when the secondary data reloads to the primary memory then the NameNode main memory limitation arises again. The focus of this research is to reduce the namespace problem of main memory and to make the system dynamically scalable. A new Metadata Fragmentation Algorithm is proposed that separates the metadata list of NameNode dynamically. The NameNode creates Secondary Memory File in perspective of the threshold value and allocates secondary memory location based on the requirement. According to the proposed algorithm the maximum third, out of fourth of main memory is used at the secondary file caching time. The free space aids in faster operation by Dynamically Scalable NameNode approach. This proposed algorithm shows that the space utilization is increased to 17% and time utilization is increased to 0.0005% with the comparison of the existing fragmentation algorithm.

show abstract

“…A classification based metadata management system is proposed in [2]. They focused on reducing the bottleneck of the NameNode main memory.…”

Section: Literature Reviewmentioning

confidence: 99%

A noble approach to develop dynamically scalable namenode in hadoop distributed file system using secondary storage

Shaha¹,

Akhtar²,

Johora³

et al. 2019

IJEECS

View full text Add to dashboard Cite

show abstract

“…The current management method on cloud features a single centralized node that is dedicated to respond all the data request from brokers, referred as name-node [31]. This kind of data management has two drawbacks that 1) the name-node has to trace the changes and movements of all PHR data with high CPU and networking utilities, and 2) inquiring data location in a single big mapping table is a time-consuming task.…”

Section: Data Coherence Protocol In Cloudmentioning

confidence: 99%

Enabling real-time information service on telehealth system over cloud-based big data platform

Wang

Qiu

Guo

2017

Journal of Systems Architecture

View full text Add to dashboard Cite

A telehealth system covers both clinical and nonclinical uses, which not only provides store-and-forward data services to be offline studied by relevant specialists, but also monitors the real-time physiological data through ubiquitous sensors to support remote telemedicine. However, the current telehealth systems don't consider the velocity and veracity of the big-data system in the medical context. Emergency events generate a large amount of the real-time data, which should be stored in the data center, and forwarded to remote hospitals. Furthermore, patients' information is scattered on the distributed data center, which cannot provide a high-efficient remote real-time service. In this paper, we proposes a probability-based bandwidth model in a telehealth cloud system, which helps cloud broker to provide a high performance allocation of computing nodes and links. This brokering mechanism considers the location protocol of Personal Health Record (PHR) in cloud and schedules the real-time signals with a low information transfer between different hosts. The broker uses several bandwidth evaluating methods to predict the near future usage of bandwidth in a telehealth context. The simulation results show that our model is effective at determining the best performing service, and the inserted service validates the utility of our approach.

show abstract

“…According to the storage and processing of the metadata [20] [21] in Hadoop, the Area of each data block and the number of key columns that satisfied the current Area are recorded during importation: this information is stored by generating an attached table named DB_TABLENAME_META. Here, DB_TABLENAME_META mainly contained three fields: BlockID, Area, and KNum.…”

Section: Storage and Application Of Metadatamentioning

confidence: 99%

An Efficient Distributed Data Management Method based key Columns Partition Preprocessing

Zhang²,

Li³

2015

IJDTA

View full text Add to dashboard Cite

show abstract

Classification Based Metadata Management for HDFS

Cited by 5 publications

References 4 publications

A noble approach to develop dynamically scalable namenode in hadoop distributed file system using secondary storage

A noble approach to develop dynamically scalable namenode in hadoop distributed file system using secondary storage

Enabling real-time information service on telehealth system over cloud-based big data platform

An Efficient Distributed Data Management Method based key Columns Partition Preprocessing

Contact Info

Product

Resources

About