Extracting Clean Performance Models from Tainted Programs

Copik, Marcin; Calotoiu, Alexandru; Grosser, Tobias; Wicki, Nicolas; Wolf, Felix; Hoefler, Torsten

doi:10.48550/arxiv.2012.15592

Cited by 2 publications

(2 citation statements)

References 37 publications

(55 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As mentioned earlier, we rely on a hybrid of analytical modeling and empirical parametrization for ParaDL. To reduce the impact of noise associated with black-box empirical modeling [8], we segment the experiments used to inspect the target parameters. We are thus able to distinguish between effects of noise on the measurements and actual runtime change because of parameter influence.…”

Section: Empirical Parametrizationmentioning

confidence: 99%

An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks

Kahira

Nguyen

Bautista-Gomez

et al. 2021

Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing

View full text Add to dashboard Cite

Deep Neural Network (DNN) frameworks use distributed training to enable faster time to convergence and alleviate memory capacity limitations when training large models and/or using high dimension inputs. With the steady increase in datasets and model sizes, model/hybrid parallelism is deemed to have an important role in the future of distributed training of DNNs. We analyze the compute, communication, and memory requirements of Convolutional Neural Networks (CNNs) to understand the trade-offs between different parallelism approaches on performance and scalability. We leverage our model-driven analysis to be the basis for an oracle utility which can help in detecting the limitations and bottlenecks of different parallelism approaches at scale. We evaluate the oracle on six parallelization strategies, with four CNN models and multiple datasets (2D and 3D), on up to 1024 GPUs. The results demonstrate that the oracle has an average accuracy of about 86.74% when compared to empirical results, and as high as 97.57% for data parallelism. CCS CONCEPTS• Computing methodologies → Parallel computing methodologies; Distributed computing methodologies; Machine learning.

show abstract

Section: Empirical Parametrizationmentioning

confidence: 99%

An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks

Kahira

Nguyen

Bautista-Gomez

et al. 2021

Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing

View full text Add to dashboard Cite

show abstract

“…We additionally highlight parallel work on improving the cost and accuracy of empirical performance modeling that utilizes compiler-based analysis to selectively execute loop iterations. [50]. Our work is set apart from this previous autotuning research by the use of online profiling, selective execution of subroutines, and distributed-memory critical-path analysis.…”

Section: B Autotuning Mechanismsmentioning

confidence: 99%

Accelerating Distributed-Memory Autotuning via Statistical Analysis of Execution Paths

Hutter¹,

Solomonik²

2021

Preprint

View full text Add to dashboard Cite

The prohibitive expense of automatic performance tuning at scale has largely limited the use of autotuning to libraries for shared-memory and GPU architectures. We introduce a framework for approximate autotuning that achieves a desired confidence in each algorithm configuration's performance by constructing confidence intervals to describe the performance of individual kernels (subroutines of benchmarked programs). Once a kernel's performance is deemed sufficiently predictable for a set of inputs, subsequent invocations are avoided and replaced with a predictive model of the execution time. We then leverage online execution path analysis to coordinate selective kernel execution and propagate each kernel's statistical profile. This strategy is effective in the presence of frequentlyrecurring computation and communication kernels, which is characteristic to algorithms in numerical linear algebra. We encapsulate this framework as part of a new profiling tool, Critter, that automates kernel execution decisions and propagates statistical profiles along critical paths of execution. We evaluate performance prediction accuracy obtained by our selective execution methods using state-of-the-art distributedmemory implementations of Cholesky and QR factorization on Stampede2, and demonstrate speed-ups of up to 7.1x with 98% prediction accuracy.

show abstract

Extracting Clean Performance Models from Tainted Programs

Cited by 2 publications

References 37 publications

An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks

An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks

Accelerating Distributed-Memory Autotuning via Statistical Analysis of Execution Paths

Contact Info

Product

Resources

About