Errata for "Cerebro: a data system for optimized deep learning model selection"

Nakandala, Supun; Zhang, Yuhao; Kumar, Arun

doi:10.14778/3447689.3447691

Cited by 15 publications

(22 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Coordinating HP search jobs must be done carefully to ensure that: each job processes the entire dataset exactly once per epoch. A naive way of doing this is to pre-process the dataset once and reuse across all HP jobs and all epochs as suggested by prior work [58,72]. This approach will not work for two reasons.…”

Section: Coordinated Prepmentioning

confidence: 99%

See 1 more Smart Citation

Analyzing and Mitigating Data Stalls in DNN Training

Mohan¹,

Phanishayee²,

Raniwala³

et al. 2020

Preprint

View full text Add to dashboard Cite

Training Deep Neural Networks (DNNs) is resource-intensive and time-consuming. While prior research has explored many different ways of reducing DNN training time spanning different layers of the system stack, the impact of input data pipeline, i.e., fetching raw data items from storage and performing data pre-processing in memory, has been relatively unexplored.This paper makes the following contributions: (1) We present the first comprehensive analysis of how the input data pipeline affects the training time of widely-used Deep Neural Networks (DNNs). We analyze nine different models across three tasks and four datasets while varying factors such as the amount of memory, number of CPU threads, storage device, GPU generation etc on servers that are a part of a large production cluster at Microsoft. We find that in many cases, DNN training time is dominated by data stall time: time spent waiting for data to be fetched and pre-processed. ( 2) We build a tool, DS-Analyzer to precisely measure data stalls using a differential technique, and perform predictive what-if analysis on data stalls. (3) Finally, based on the insights from our analysis, we design and implement three simple but effective techniques in a data-loading library, CoorDL, to mitigate data stalls. Our experiments on a range of DNN tasks, models, datasets, and hardware configurations show that when PyTorch uses CoorDL instead of the state-of-the-art DALI data loading library, DNN training time is reduced significantly (by as much as 5× on a single server).

show abstract

Section: Coordinated Prepmentioning

confidence: 99%

“…This work therefore fundamentally improves performance on top of what Quiver can achieve. While prior work like Cerebro [72], and DeepIO [103] have looked at optimizing data fetch in distributed training, they do not systematically analyze data stalls in different training scenarios, or demonstrate how to accelerate single-server training.…”

Section: Introductionmentioning

confidence: 99%

Analyzing and Mitigating Data Stalls in DNN Training

Mohan¹,

Phanishayee²,

Raniwala³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…In addition, since convolutional layers are the most difficult type of layers for efficient recomputation, our solution focuses on convolutional layers to provide high compression ratios with minimum performance overheads and accuracy losses. We also note that COMET can further improve the training performance by combining with model parallelism techniques such as Cerebro [39], which is designed for efficiently training multiple model configurations to select the best model configuration.…”

Section: Training Large-scale Dnnsmentioning

confidence: 99%

A novel memory-efficient deep learning training framework via error-bounded lossy compression

Jin

Song

et al. 2021

Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming

View full text Add to dashboard Cite

Deep neural networks (DNNs) are becoming increasingly deeper, wider, and non-linear due to the growing demands on prediction accuracy and analysis quality. Training wide and deep neural networks require large amounts of storage resources such as memory because the intermediate activation data must be saved in the memory during forward propagation and then restored for backward propagation. However, state-of-the-art accelerators such as GPUs are only equipped with very limited memory capacities due to hardware design constraints, which significantly limits the maximum batch size and hence performance speedup when training large-scale DNNs. Traditional memory saving techniques either suffer from performance overhead or are constrained by limited interconnect bandwidth or specific interconnect technology.In this paper, we propose a novel memory-efficient CNN training framework (called COMET) that leverages error-bounded lossy compression to significantly reduce the memory requirement for training in order to allow training larger models or to accelerate training. Different from the state-of-the-art solutions that adopt image-based lossy compressors (such as JPEG) to compress the activation data, our framework purposely adopts error-bounded lossy compression with a strict error-controlling mechanism. Specifically, we perform a theoretical analysis on the compression error propagation from the altered activation data to the gradients, and empirically investigate the impact of altered gradients over the training process. Based on these analyses, we optimize the errorbounded lossy compression and propose an adaptive error-bound control scheme for activation data compression. We evaluate our design against state-of-the-art solutions with five widely-adopted CNNs and ImageNet dataset. Experiments demonstrate that our proposed framework can significantly reduce the training memory consumption by up to 13.5× over the baseline training and 1.8× over another state-of-the-art compression-based framework, respectively, with little or no accuracy loss.

show abstract

“…Moving data between workers incurs expensive network traffic and is not viable. Instead, existing solutions use different learning rates across workers [24] or train multiple models concurrently and choose the one that provides the highest accuracy [36]. CROSSBOW [27] considers heterogeneity in the context of multiple GPUs by scheduling a different number of processing streams -learners -on every GPU.…”

Section: Elastic Distributed Trainingmentioning

confidence: 99%

Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers

Ma,

Rusu,

et al. 2021

Preprint

View full text Add to dashboard Cite

Motivated by extreme multi-label classification applications, we consider training deep learning models over sparse data in multi-GPU servers. The variance in the number of non-zero features across training batches and the intrinsic GPU heterogeneity combine to limit accuracy and increase the time to convergence. We address these challenges with Adaptive SGD, an adaptive elastic model averaging stochastic gradient descent algorithm for heterogeneous multi-GPUs that is characterized by dynamic scheduling, adaptive batch size scaling, and normalized model merging. Instead of statically partitioning batches to GPUs, batches are routed based on the relative processing speed. Batch size scaling assigns larger batches to the faster GPUs and smaller batches to the slower ones, with the goal to arrive at a steady state in which all the GPUs perform the same number of model updates. Normalized model merging computes optimal weights for every GPU based on the assigned batches such that the combined model achieves better accuracy. We show experimentally that Adaptive SGD outperforms four state-of-the-art solutions in time-to-accuracy and is scalable with the number of GPUs.

show abstract

Errata for "Cerebro: a data system for optimized deep learning model selection"

Cited by 15 publications

References 1 publication

Analyzing and Mitigating Data Stalls in DNN Training

Analyzing and Mitigating Data Stalls in DNN Training

A novel memory-efficient deep learning training framework via error-bounded lossy compression

Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers

Contact Info

Product

Resources

About