A generic communication scheduler for distributed DNN training acceleration

Network-accelerated distributed machine learning for multi-tenant settings

“…E.g., ResNet-50 and ResNet-152 [25] models that achieve up to 75% accuracy in classifying images [19] are 100MB and 240MB in size respectively. For such large models, it is well known [39] that the overall training time is dominated by communication time taken to share updates among many parallel workers; for ResNet-50 model trained with 30 GPUs (NVIDIA P100), we observe that per-iteration communication time (320 ) is 3× the gradient computation time (100 ).…”

“…Algorithms: We evaluate the following: PS-based asynchronous and synchronous variants of MLfabric, or MLfabric-A and MLfabric-S, respectively; vanilla PS-based asynchronous (Async); and AllReduce-based (using NCCL library) synchronous algorithms using ring-reduce communication (RR-Sync). We also compare with other state-of-the-art approaches: (1) SwitchML [42], where aggregation of gradient updates happen on P4 [37] network switches as opposed to end hosts, and (2) BytePS [39], an alternate communication library based on parameter server architecture.…”

“…Comparison with other frameworks: We also compare MLfabric-S with SwitchML [42] and BytePS [39] which leverage switch-level aggregation, and scheduling without aggregation, respectively. We train a ResNet-152 model using 8 GPUs on 8 nodes connected by 10Gbps ethernet (no RDMA); we chose this configuration (different from the rest of our experiments) since SwitchML did not support multiple GPUs on a single node.…”

“…Per-iteration runtime in BytePS is determined by the slowest server. Because of Tensor partitioning [39], BytePS is able to efficiently pipeline update transfers; thus, the slowdown is less severe (but still 2×).…”

“…DML algorithms, e.g., stochastic gradient descent (SGD), and Latent Dirichlet Allocation (LDA) [10,32], are iterative in nature, and are both computation and communication intensive ( §2). Over the years, a variety of DML systems [1,6,7,17] were developed to accelerate training by improving worker computation, e.g., via hardware accelerators [23,38], improving communication efficiency [2,3,42,48], and coordinating computation with communication [24,39,49,55]. These systems generally use one of two architectures: (a) Parameter Server (PS) [12,50], where, the model is stored at a separate location (server); in every iteration workers pull the latest model and compute an update, which is then shipped to the server and applied to the model.…”