Speeding up Deep Learning with Transient Servers

Abstract

Distributed training frameworks, like TensorFlow, have been proposed as a means to reduce the training time of deep learning models by using a cluster of GPU servers. While such speedups are often desirable-e.g., for rapidly evaluating new model designs-they often come with significantly higher monetary costs due to sublinear scalability. In this paper, we investigate the feasibility of using training clusters composed of cheaper transient GPU servers to get the benefits of distributed training without the high costs.We conduct the first large-scale empirical analysis, launching more than a thousand GPU servers of various capacities, aimed at understanding the characteristics of transient GPU servers and their impact on distributed training performance. Our study demonstrates the potential of transient servers with a speedup of 7.7X with more than 62.9% monetary savings for some cluster configurations. We also identify a number of important challenges and opportunities for redesigning distributed training frameworks to be transient-aware. For example, the dynamic cost and availability characteristics of transient servers suggest the need for frameworks to dynamically change cluster configurations to best take advantage of current conditions.

Elastic AI: system support for adaptive machine learning in pervasive computing systems

“…As a third group of related work, there are a number of systems, most prominently in the domain of the Internet of Things, that are specifically focusing on runtime support for AI components in networked systems, e.g. Amazon (2021), Kim and Kim (2020), Li et al (2019) and Microsoft (2021). However, these focus purely on running AI models in the Cloud or Edge and do not support the level of dynamism necessary for pervasive application cases.…”

Characterizing and Modeling Distributed Training with Transient Cloud GPU Servers

“…One of the key challenges of using transient servers for distributed training is that they can be revoked at any time. Even the revocation of a single worker can lead to significant performance degradation [7]. In this section, we characterize the revocation patterns of Google Cloud's transient servers.…”

“…Popular deep learning frameworks [6,[27][28][29] provide distributed SGD-based algorithms [30,31] to train increasingly bigger models on larger datasets. Existing works towards understanding distributed training workloads can be broadly categorized into performance modeling [1,9,10] and empirical studies [1,7,22,32,33]. In contrast to prior model-driven performance modeling studies [10,12,13], where a static endto-end training time prediction is the main focus, our work leverages data-driven modeling that is powered by a largescale empirical measurement in a popular cloud platform.…”

“…Additionally, prior work also accounted for application-specific requirements, such as interactivity, when designing transient-aware mechanisms [38,41]. As a promising and cheap way to provide good parallelisms, transient servers have garnered a lot of interests for big data big data analytics [39,40,42,43], memory-intensive applications [44], cluster resource managers [45,46], and most recently deep learning [7]. Our work provides a new perspective with a focus on characterizing and modeling distributed training on transient servers.…”

“…Revoked GPU servers often mean significant loss of work and require manual effort by the practitioner to request new servers, to reconfigure the training cluster, and even to diagnose potential performance bottlenecks. Concretely, when a GPU server is revoked, all its local training progress will disappear and in the worst case, the revocation will also impede the functionality of saving the trained model [6,7].…”