Performance Optimization System for Hadoop and Spark Frameworks

Astsatryan, Hrachya; Kocharyan, Aram; Hagimont, Daniel; Lalayan, Arthur

doi:10.2478/cait-2020-0056

Cited by 10 publications

(8 citation statements)

References 25 publications

(19 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Where fluxnum is a function implementing the numerical flux and N is the array of the normal vectors of the cells defined in (3).…”

Section: Transport Simulationmentioning

confidence: 99%

See 1 more Smart Citation

Reducing the memory usage of Lattice-Boltzmann schemes with a DWT-based compression

Flint¹,

Helluy²

2023

Preprint

View full text Add to dashboard Cite

This paper presents a new solution to address the challenge of increasing memory usage in high-performance computing simulations of Lattice-Bolzmann or Finite-Volume schemes. Our approach utilises a lossy compression scheme based on the Discrete Wavelet Transform (DWT) to achieve high compression ratios while preserving the accuracy of the simulation. Our evaluation on two different FV/LBM schemes demonstrates that the approach can reduce memory usage by several orders of magnitude.Résumé. Ce papier présente une nouvelle solution pour faire face à l'augmentation de l'utilisation de la mémoire dans les simulations haute performance basées sur les méthodes Lattice-Bolzmann ou Volumes Finis. Notre approche utilise un schéma de compression avec perte basé sur la transformée par ondelettes discrète (DWT) pour obtenir des taux de compression élevés tout en préservant la précision de la simulation. Notre évaluation sur deux différents schémas VF/LBM démontre que l'approche peut réduire l'utilisation de la mémoire de plusieurs ordres de grandeur.

show abstract

“…Where fluxnum is a function implementing the numerical flux and N is the array of the normal vectors of the cells defined in (3).…”

Section: Transport Simulationmentioning

confidence: 99%

“…The use of data compression in the field of high-performance computing has already been explored in several works [3,5]. Besides memory saving, the use of data compression can also be motivated by the need to transfer data between the CPU and the GPU efficiently.…”

Section: Introductionmentioning

confidence: 99%

Reducing the memory usage of Lattice-Boltzmann schemes with a DWT-based compression

Flint¹,

Helluy²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The data compression techniques can reduce storage usage and the number of I/O operations, improving processing performance. Recent studies [14,15] show that compression methods combined with HPC can significantly enhance the performance of Big data workflows. One of the optimal satellite image formats is Cloud Optimized GeoTIFF (COG) [16], which provides essential advantages compared to traditional formats, such as NetCDF [17].…”

Section: Background and Motivationmentioning

confidence: 99%

Data Compression-Aware Performance Analysis of Dask and Spark for Earth Observation Data Processing

Lalayan¹

2023

MPCS

View full text Add to dashboard Cite

High-performance computing is a good choice for handling Big Earth Observation data, allowing the processing of the data in a distributed and performance-efficient way using in-memory computing frameworks. The data compression technique reduces the amount of storage and network transfer time and improves processing performance. The article aims to investigate the effectiveness of widely used distributed data processing frameworks in conjunction with lossless data compression techniques, to find the optimal compression method and processing framework for specific earth observation workflows. Normalized Difference Vegetation Index has been evaluated for the territory of Armenia, obtaining data from the Sentinel satellite and considering the supported compression methods to compare the performance of in-memory Dask and Spark frameworks. Experiments show that the Zstandard compression method and the Dask framework are the best choices for such workflows.

show abstract

“…The article presents a codeless performance-efficient decision-making system mixing the approaches mentioned above to enhance our previous studies [8].…”

Section: Related Workmentioning

confidence: 99%

“…Before processing data, the framework decompresses the data if the input file is compressed in HDFS. Our recent studies show [7,8,9] that the average memory usage for selected scientific workflows is 13-17% for Hadoop and 20-40% for Spark jobs, which neglect the full utilization of the RAM of HDFS nodes. Therefore, the usage of RAM-free space may boost the performance of HDFS processing.…”

Section: Introductionmentioning

confidence: 99%

Performance-efficient Recommendation and Prediction Service for Big Data frameworks focusing on Data Compression and In-memory Data Storage Indicators

Astsatryan

Lalayan

Kocharyan³

et al. 2021

SCPE

Self Cite

View full text Add to dashboard Cite

The MapReduce framework manages Big Data sets by splitting the large datasets into a set of distributed blocks and processes them in parallel. Data compression and in-memory file systems are widely used methods in Big Data processing to reduce resource-intensive I/O operations and improve I/O rate correspondingly. The article presents a performance-efficient modular and configurable decision-making robust service relying on data compression and in-memory data storage indicators. The service consists of Recommendation and Prediction modules, predicts the execution time of a given job based on metrics, and recommends the best configuration parameters to improve Hadoop and Spark frameworks' performance. Several CPU and data-intensive applications and micro-benchmarks have been evaluated to improve the performance, including Log Analyzer, WordCount, and K-Means.

show abstract

Performance Optimization System for Hadoop and Spark Frameworks

Cited by 10 publications

References 25 publications

Reducing the memory usage of Lattice-Boltzmann schemes with a DWT-based compression

Reducing the memory usage of Lattice-Boltzmann schemes with a DWT-based compression

Data Compression-Aware Performance Analysis of Dask and Spark for Earth Observation Data Processing

Performance-efficient Recommendation and Prediction Service for Big Data frameworks focusing on Data Compression and In-memory Data Storage Indicators

Contact Info

Product

Resources

About