Analyzing and Mitigating Data Stalls in DNN Training

Abstract

Training Deep Neural Networks (DNNs) is resource-intensive and time-consuming. While prior research has explored many different ways of reducing DNN training time spanning different layers of the system stack, the impact of input data pipeline, i.e., fetching raw data items from storage and performing data pre-processing in memory, has been relatively unexplored.This paper makes the following contributions: (1) We present the first comprehensive analysis of how the input data pipeline affects the training time of widely-used Deep Neural Networks (DNNs). We analyze nine different models across three tasks and four datasets while varying factors such as the amount of memory, number of CPU threads, storage device, GPU generation etc on servers that are a part of a large production cluster at Microsoft. We find that in many cases, DNN training time is dominated by data stall time: time spent waiting for data to be fetched and pre-processed. ( 2) We build a tool, DS-Analyzer to precisely measure data stalls using a differential technique, and perform predictive what-if analysis on data stalls. (3) Finally, based on the insights from our analysis, we design and implement three simple but effective techniques in a data-loading library, CoorDL, to mitigate data stalls. Our experiments on a range of DNN tasks, models, datasets, and hardware configurations show that when PyTorch uses CoorDL instead of the state-of-the-art DALI data loading library, DNN training time is reduced significantly (by as much as 5× on a single server).

Plumber: Diagnosing and Removing Performance Bottlenecks in Machine Learning Data Pipelines

“…An input bottleneck occurs when the input pipeline is not able to generate batches of training examples as fast as the training computation can consume them. If the time spent waiting for the input pipeline exceeds tens of microseconds on average, the input pipeline is not keeping up with model training, causing a data stall (Mohan et al, 2020) The current practice of pipeline tuning, which optimizes the throughput (rate) of the pipeline, is explained below.…”

“…Dataset Echoing (Choi et al, 2019) repeats input pipeline operations to match the rate of input pipeline with compute steps. DS-Analyzer predicts how much file cache memory is necessary to match the compute steps (Mohan et al, 2020). Progressive Compressed Records (Kuchnik et al, 2019) match compression levels to likewise minimize I/O.…”

“…Plumber generates similar plots using Dataset and resource limits. Recent studies have analyzed ML workloads and found that data pipelines can become bottlenecks (Mohan et al, 2020;Murray et al, 2021).…”

Lobster: Load Balance-Aware I/O for Distributed DNN Training

“…This pipeline overlaps with the training itself, aiming to improve resource utilization and hide the overhead of assembling the minibatches. However, despite this overlap, data loading and preprocessing often become a bottleneck [8,19,26,30,39], with reports of overheads of up to 72% of end-to-end training time [26,30]. With ever-increasing accumulation of training data, data loading is likely to become yet more costly, prompting the need for scalable solutions to mitigate these overheads.…”

Characterizing and Taming Resolution in Convolutional Neural Networks

“…Storage capacity and bandwidth requirements (bytes moved from storage to compute device per image) also scale quadratically with image resolution, affecting the monetary cost (both storage and network usage are billed) of inference in real-world datacenter or cloud deployments where a separate storage cluster is usually used to store and forward input data through the network [21]. As a result, DNN training is frequently dominated by data stall time, which happens both remotely and locally, and can be due to CPU decoding overhead [20].…”