Network-accelerated distributed machine learning for multi-tenant settings

Abstract

Many distributed machine learning (DML) workloads are increasingly being run in shared clusters. Training in such clusters can be impeded by unexpected compute and network contention, resulting in stragglers. We present MLfabric, a contention-aware DML system that manages the performance of a DML job running in a shared cluster. The DML application hands all network communication (gradient and model transfers) to the MLfabric communication library. MLfabric then carefully orders transfers to improve convergence, opportunistically aggregates them at idle DML workers to improve resource efficiency, and replicates them to support new notions of fault tolerance, while systematically accounting for compute stragglers and network contention. We find that MLfabric achieves up to 3× speed-up in training large deep learning models in realistic dynamic cluster settings. CCS CONCEPTS • Computing methodologies → Parallel algorithms.

Constrained In-network Computing with Low Congestion in Datacenter Networks

“…Specifically for such ML tasks, network performance has been noted as a major bottleneck hindering the efficient usage of such frameworks [3], [35]. Various approaches have been suggested to modify ML methodologies in order to improve upon the network induced performance of distributed ML [4], [36], [37].…”

“…As online applications and services increase in popularity, distributed data processing capabilities and datacenter networks have become a major part of the infrastructure of modern society. Moreover, due to the vast growth in the amount of data processed by such applications, recent work shows that the bottleneck for efficient distributed computation is now the underlying communication network and not the computational capabilities at the servers [1]- [3], as was traditionally the case.…”

SOAR: Minimizing Network Utilization with Bounded In-network Computing

“…Efficiently performing distributed machine learning, and specifically the task of training deep neural networks, has been a fundamental concern in the past decade. In particular, network bottlenecks are arguably one of the major concerns when executing such tasks [29,53]. Various methods for improving network performance and footprint in such systems have been proposed and implemented, including sparsification, quantization, and scheduling [17,54,58].…”

“…Datacenter networks and their distributed data processing capabilities are the driving force behind leading applications and services, including search engines, content distribution, social networks and eCommerce. Recent work has shown that for many of the tasks performed by such applications, the network (and not server computation) is the actual bottleneck hindering the ability to optimize computation efficiency and performance [13,35,53]. Such bottlenecks occur, e.g., in distributed and federated machine learning (e.g., AllReduce), and in solutions employing the MapReduce methodology for big data tasks, and more generally in scenarios giving rise to the incast problem [5,56].…”