Large-scale compute-intensive analysis via a combined in-situ and co-scheduling workflow approach

Sewell, Christopher; Heitmann, Katrin; Finkel, Hal; Zagaris, George; Parete-Koon, Suzanne; Fasel, Patricia; Pope, Adrian; Frontiere, Nicholas; Lo, Li-Ta; Messer, O. E. Bronson; Habib, Salman; Ahrens, James

doi:10.1145/2807591.2807663

Cited by 28 publications

(17 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the in-situ approach, simulation and analyzes are co-located in the same node, while in the in-transit approach, the data analyzes are outsourced onto dedicated nodes [12]. Several studies have shown that large-scale simulations with in-situ could benefit from co-scheduling approaches [11,15]. The difficulty consists in ensuring that the in-situ part processes the data fast enough to avoid slowing down the main simulation, which is directly related to co-scheduling issues: how to partition the resources across the concurrent analysis applications that share the CMP?…”

Section: Related Workmentioning

confidence: 99%

“…In this paper, we also use CAT to partition the LLC into several areas when coscheduling applications, but with the objective of optimizing the throughput of in-situ or in-transit analysis for large-scale simulations. Indeed, in such simulations, data is generated at each iteration and periodically analyzed by parallel processes on dedicated nodes, concurrently of the main simulation [11]. If these dedicated nodes belong to the main simulation platform (thereby reducing the number of available cores for simulation), we speak of in-situ processing, while if they belong to an auxiliary platform, we speak of of in-transit processing [12].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Co-Scheduling HPC Workloads on Cache-Partitioned CMP Platforms

Aupy

Benoît

Goglin

et al. 2018

2018 IEEE International Conference on Cluster Computing (CLUSTER)

View full text Add to dashboard Cite

With the recent advent of many-core architectures such as chip multiprocessors (CMP), the number of processing units accessing a global shared memory is constantly increasing. Co-scheduling techniques are used to improve application throughput on such architectures, but sharing resources often generates critical interferences. In this paper, we focus on the interferences in the last level of cache (LLC) and use the Cache Allocation Technology (CAT) recently provided by Intel to partition the LLC and give each co-scheduled application their own cache area. We consider m iterative HPC applications running concurrently and answer to the following questions: (i) how to precisely model the behavior of these applications on the cache partitioned platform? and (ii) how many cores and cache fractions should be assigned to each application to maximize the platform efficiency? Here, platform efficiency is defined as maximizing the performance either globally, or as guaranteeing a fixed ratio of iterations per second for each application. Through extensive experiments using CAT, we demonstrate the impact of cache partitioning when multiple HPC application are co-scheduled onto CMP platforms.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Co-Scheduling HPC Workloads on Cache-Partitioned CMP Platforms

Aupy

Benoît

Goglin

et al. 2018

2018 IEEE International Conference on Cluster Computing (CLUSTER)

View full text Add to dashboard Cite

show abstract

“…Scientific data that generated from large‐scale simulations and observational devices are growing rapidly in data size and complexity. A fundamental challenge of managing scientific data on high‐performance computing (HPC) systems is to provide efficient data locating services in both file level and record level.…”

Section: Introductionmentioning

confidence: 99%

UniIndex: An index and query middleware for parallel file systems

Cheng

Wang

et al. 2019

Concurrency and Computation

View full text Add to dashboard Cite

Summary As data analysis scenarios keep increasing on high‐performance computing systems, the ability to select a small fraction of data from a large volume of scientific data sets is vital to accelerate scientific discovery. However, parallel file systems lack the ability to provide efficient data locating services at the granularity of both a file and a record. Existing methods for identifying and indexing data are often domain‐specific and do not scale to large scientific data sets. In this paper, we describe the design and implementation of UniIndex framework, which combines our proposed techniques for user‐annotation extraction, in‐memory cache layer, in‐situ indexing, and parallel query processing. Acting as middleware on top of production file systems, UniIndex enables efficient data locating services with minimal user effort. Our evaluations show that UniIndex can locate target files from directories containing millions of files in microseconds. By applying in situ indexing and the lightweight range‐bitmap index, record‐level index building time can be dramatically reduced while maintaining up to two orders of magnitude query speedup than scanning the entire data set.

show abstract

“…These solutions are starting to be implemented for HPC applications. Sewell et al [27] explain that in the case of the HACC application (a cosmological code), petabytes of data are created to be analyzed later. The analysis is done by multiple independent processes.…”

Section: Introductionmentioning

confidence: 99%

Co-Scheduling Algorithms for Cache-Partitioned Systems

Aupy

Benoît

Pottier

et al. 2017

2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)

View full text Add to dashboard Cite

Cache-partitioned architectures allow subsections of the shared last-level cache (LLC) to be exclusively reserved for some applications. This technique dramatically limits interactions between applications that are concurrently executing on a multicore machine. Consider n applications that execute concurrently, with the objective to minimize the makespan, defined as the maximum completion time of the n applications. Key scheduling questions are: (i) which proportion of cache and (ii) how many processors should be given to each application? Here, we assign rational numbers of processors to each application, since they can be shared across applications through multi-threading. In this paper, we provide answers to (i) and (ii) for perfectly parallel applications. Even though the problem is shown to be NP-complete, we give key elements to determine the subset of applications that should share the LLC (while remaining ones only use their smaller private cache). Building upon these results, we design efficient heuristics for general applications. Extensive simulations demonstrate the usefulness of co-scheduling when our efficient cache partitioning strategies are deployed.

show abstract

Large-scale compute-intensive analysis via a combined in-situ and co-scheduling workflow approach

Cited by 28 publications

References 29 publications

Co-Scheduling HPC Workloads on Cache-Partitioned CMP Platforms

Co-Scheduling HPC Workloads on Cache-Partitioned CMP Platforms

UniIndex: An index and query middleware for parallel file systems

Co-Scheduling Algorithms for Cache-Partitioned Systems

Contact Info

Product

Resources

About