Big Data, Simulations and HPC Convergence

Fox, Geoffrey; Qiu, Judy; Jha, Shantenu; Ekanayake, Saliya; Kamburugamuve, Supun

doi:10.1007/978-3-319-49748-8_1

Cited by 40 publications

(32 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…To foster faster model convergence, we need to design new collective communication abstractions. We identify all 5 classes of data-intensive computation [2], from pleasingly parallel to machine learning and simulations. To re-design a modular software stack with native kernels to effectively utilize scale-up servers for machine learning and data analytics applications.…”

Section: A Hpc For Machine Learningmentioning

confidence: 99%

See 1 more Smart Citation

Learning Everywhere: Pervasive Machine Learning for Effective High-Performance Computation

Fox

Glazier

Kadupitiya

et al. 2019

2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)

Self Cite

View full text Add to dashboard Cite

The convergence of HPC and data intensive methodologies provide a promising approach to major performance improvements. This paper provides a general description of the interaction between traditional HPC and ML approaches and motivates the "Learning Everywhere" paradigm for HPC. We introduce the concept of "effective performance" that one can achieve by combining learning methodologies with simulation based approaches, and distinguish between traditional performance as measured by benchmark scores. To support the promise of integrating HPC and learning methods, this paper examines specific examples and opportunities across a series of domains. It concludes with a series of open computer science and cyberinfrastructure questions and challenges that the Learning Everywhere paradigm presents.• HPCforML: Using HPC to execute and enhance ML performance, or using HPC simulations to train ML algorithms (theory guided machine learning), which are then used to understand experimental data or simulations.

show abstract

Section: A Hpc For Machine Learningmentioning

confidence: 99%

“…The convergence of HPC and data-intensive methodologies [2] provide a promising approach to major performance improvements. Traditional HPC simulations are reaching the limits of original progress.…”

Section: Introductionmentioning

confidence: 99%

Learning Everywhere: Pervasive Machine Learning for Effective High-Performance Computation

Fox

Glazier

Kadupitiya

et al. 2019

2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The convergence between Big Data and HPC can benefit to businesses, willing to dedicate their HPC environments (or HPC‐like environments, also called enterprise performance computing) to several purposes such as deep learning, artificial intelligence, and Big Data. Researchers and industries already started to exploit this convergence . As an example of a practical case, Kamburugamuve et al exploited HPC high‐performance interconnects, namely, Infiniband and Intel Omni‐Path, in order to improve the performance of the Apache Heron Big Data framework.…”

Section: Introductionmentioning

confidence: 99%

Big data deployment in containerized infrastructures through the interconnection of network namespaces

Sauvanaud

Dholakia²,

Guitart

et al. 2020

Softw Pract Exp

View full text Add to dashboard Cite

Big Data applications tackle the challenge of fast handling of large streams of data. Their performance is not only dependent on the data frameworks implementation and the underlying hardware but also on the deployment scheme and its potential for fast scaling. Consequently, several efforts have focused on the ease of deployment of Big Data applications, notably through the use of containerization. This technology was indeed raised to bring multitenancy and multiprocessing out of clusters, providing high deployment flexibility through lightweight container images. Recent studies have focused mostly on Docker containers. Notwithstanding, this article is actually interested in recent Singularity containers as they provide more security and support high-performance computing (HPC) environments and, in this way, they can make Big Data applications benefit from the specialized hardware of HPC. Singularity 2.x, however, does not isolate network resources as required by most Big Data components.Singularity 3.x allows allocating each container with isolated network resources, but their interconnection requires a nontrivial amount of configuration effort. In this context, this article makes a functional contribution in the form of a deployment scheme based on the interconnection of network namespaces, through underlay and overlay networking approaches, to make Big Data applications easily deployable inside Singularity containers. We provide detailed account of our deployment scheme when using both interconnection approaches in the form of a "how-to-do-it" report, and we evaluate it by comparing three Big Data applications based on Hadoop when performing on a bare-metal infrastructure and on scenarios involving Singularity and Docker instances. K E Y W O R D S

show abstract

“…HPC workloads are traditionally dominated by parallel applications that are large, parallel and tightly-coupled. Recently, however, other application types have become more prevalent, including dramatically more data-intensive and highthroughput applications and real-time jobs [9], [10]. Therefore, jobs that share the same node can have distinct scheduling objectives and different resource requirements.…”

Section: Introductionmentioning

confidence: 99%

Improving Per-Node Computing Efficiency by an Adaptive Lock-Free Scheduling Model

Zheng

Zhou

et al. 2018

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

Job scheduling on many-core computers with tens or even hundreds of processing cores is one of the key technologies in High Performance Computing (HPC) systems. Despite many scheduling algorithms have been proposed, scheduling remains a challenge for executing highly effective jobs that are assigned in a single computing node with diverse scheduling objectives. On the other hand, the increasing scale and the need for rapid response to changing requirements are hard to meet with existing scheduling models in an HPC node. To address these issues, we propose a novel adaptive scheduling model that is applied to a single node with a many-core processor; this model solves the problems of scheduling efficiency and scalability through an adaptive optimistic control mechanism. This mechanism exposes information such that all the cores are provided with jobs and the tools necessary to take advantage of that information and thus compete for resources in an uncoordinated manner. At the same time, the mechanism is equipped with adaptive control, allowing it to adjust the number of running tools dynamically when frequent conflict happens. We justify this scheduling model and present the simulation results for synthetic and real-world HPC workloads, in which we compare our proposed model with two widely used scheduling models, i.e. multi-path monolithic and two-level scheduling. The proposed approach outperforms the other models in scheduling efficiency and scalability. Our results demonstrate that the adaptive optimistic control affords significant improvements for HPC workloads in the parallelism of the node-level scheduling model and performance.

show abstract

Big Data, Simulations and HPC Convergence

Cited by 40 publications

References 11 publications

Learning Everywhere: Pervasive Machine Learning for Effective High-Performance Computation

Learning Everywhere: Pervasive Machine Learning for Effective High-Performance Computation

Big data deployment in containerized infrastructures through the interconnection of network namespaces

Improving Per-Node Computing Efficiency by an Adaptive Lock-Free Scheduling Model

Contact Info

Product

Resources

About