Prefetching-based metadata management in Advanced Multitenant Hadoop

Won, Heesun; Son, Siwoon; Gil, Myeong-Seon; Moon, Yang-Sae

doi:10.1007/s11227-017-2019-5

Cited by 12 publications

(5 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the context of the chosen "TLC" dataset, with its multitude of features capturing diverse aspects of New York City taxi trips, VAE is ideally suited to unravel the inherent structures within this high-dimensional space. The goal is to transform the dataset into a lower-dimensional space while retaining essential information, thus mitigating the computational burden and enhancing interpretability [27].…”

Section: Balancing Reduction and Retentionmentioning

confidence: 99%

Harmonizing Dimensionality: Unveiling the Prowess of Variational Auto-Encoder in Spark for Big Data Processing

Jawad,

Al-Bakry

2024

RIA

View full text Add to dashboard Cite

In the dynamic realm of big data processing, conquering the challenges imposed by highdimensional datasets is imperative. This paper introduces a groundbreaking advancement in dimensionality reduction, employing Variational Auto-Encoder (VAE) within the Spark distributed framework. The deliberate selection of the "TLC" dataset, representative of New York City taxi trips with inherent high dimensionality, highlights the practicality of our approach. Our research showcases the virtuoso performance of VAE, achieving an impressive 95.12% reduction ratio and 89.26% accuracy. This highlights VAE's ability to elegantly distill essential information while discarding superfluous dimensions, achieving a harmonious balance between reduction and accuracy. Furthermore, building on the demonstrated superiority of Spark over Hadoop in prior successes, our adoption of VAE aligns with the overarching goal of enhancing big data processing. Spark's consistent advantage as a distributed framework reaffirms its reliability in handling diverse machine learning algorithms. This paper not only contributes to the advancement of machine learning in big data processing but also underscores the adaptability, versatility, and consistent performance of our approach across various methodologies and frameworks. The success of VAE in reducing dimensionality, coupled with Spark's inherent advantages, positions this research as a valuable contribution to the exploration of advanced techniques in distributed big data processing.

show abstract

Section: Balancing Reduction and Retentionmentioning

confidence: 99%

Harmonizing Dimensionality: Unveiling the Prowess of Variational Auto-Encoder in Spark for Big Data Processing

Jawad,

Al-Bakry

2024

RIA

View full text Add to dashboard Cite

show abstract

“…The edges are dynamically inserted or removed by the MDS to improve the prefetching accuracy. Nguyen et al [8] adopted a similar approach to mine correlations. Unfortunately, this type of prefetching approach may not be able to make correct decisions when encountering concurrent workload scenarios, particularly when there is insufficient knowledge of the access histories [11].…”

Section: Prefetching Policiesmentioning

confidence: 99%

“…Similar to the evaluation methodology described in [39], we replay a portion of each trace (i.e., the first 15,000 records) because the number of trace records is quite large. Considering that the directory entries corresponding to the files placed in the same directory are sequential [7,8], we create an empty file (we focused only on metadata access) for every logical block address (LBA) in the disk, and the filename is the same as the LBA. In other words, we convert the relevance of consecutive data blocks to the relevance of adjacent files in directories.…”

Section: Experimental Setup Of Metadata Prefetchingmentioning

confidence: 99%

See 1 more Smart Citation

Efficient Approaches for Metadata Prefetching and Data Placement in Wide-Area Computing

Wei

Xin

Zhang

et al. 2023

Preprint

View full text Add to dashboard Cite

Metadata prefetching and data placement are essential to improve access performance for wide-area network file systems. Constructing efficient approaches has been a challenge for metadata prefetching in concurrent workload scenarios and for data placement in a wide-area network. This paper proposes efficient approaches for metadata prefetching and data placement, wherein fine-grained control of prefetching policies and variable-size data fragment writing are utilized to maximize the I/O bandwidth of distributed files. The proposed metadata prefetching approach dynamically detects the dominant workload and adaptively adjusts the prefetching policy to improve metadata access performance in concurrent workload scenarios. The proposed data placement approach places the written data fragments into the local data center to improve the write performance and sends only the location information of the data fragments to the remote data center that holds the original file. Experimental results using real system traces indicate that the file system metadata and application data access times can be reduced by up to 33.5% and 17.19%, respectively, compared to that of state-of-the-art methods.

show abstract

“…The system utilizes access-content-decoupled partitioning, speculative path resolution, and an optimistic access metadata cache method to deliver high-performance metadata operations. Researchers have addressed challenging metadata management in centralized file system, such as Hadoop, by creating metadata access graphs based on historical access values, minimizing the latency [23]. There are also studies in the field of file systems that drop network overheads by reducing the operations per an RPC request.…”

Section: Related Workmentioning

confidence: 99%

Efficient I/O Merging Scheme for Distributed File Systems

Sung

2023

Symmetry

View full text Add to dashboard Cite

Recently, decentralized file systems are widely used to overcome centralized file systems’ load asymmetry between nodes and the scalability problem. Due to the lack of a metadata server, decentralized systems require more RPC requests to control metadata processing between clients and servers, which adversely impacts the I/O performance and traffic imbalance by increasing RPC latency. In this paper, we propose an efficient I/O scheme to reduce the RPC overhead in decentralized file systems. Instead of sending a single RPC request at a time, we enqueued the RPCs in the global queue and merged them into larger RPC requests, thus avoiding excessive RPC latency overheads. The experimental results showed that our scheme improves write and read performance by up to 13% and 16%, respectively, compared with those of the original.

show abstract

Prefetching-based metadata management in Advanced Multitenant Hadoop

Cited by 12 publications

References 19 publications

Harmonizing Dimensionality: Unveiling the Prowess of Variational Auto-Encoder in Spark for Big Data Processing

Harmonizing Dimensionality: Unveiling the Prowess of Variational Auto-Encoder in Spark for Big Data Processing

Efficient Approaches for Metadata Prefetching and Data Placement in Wide-Area Computing

Efficient I/O Merging Scheme for Distributed File Systems

Contact Info

Product

Resources

About