What does fault tolerant deep learning need from MPI?

Abstract

Deep Learning (DL) algorithms have become the de facto Machine Learning (ML) algorithm for large scale data analysis. DL algorithms are computationally expensive -even distributed DL implementations which use MPI require days of training (model learning) time on commonly studied datasets. Long running DL applications become susceptible to faults -requiring development of a fault tolerant system infrastructure, in addition to fault tolerant DL algorithms. This raises an important question: What is needed from MPI for designing fault tolerant DL implementations? In this paper, we address this problem for permanent faults. We motivate the need for a fault tolerant MPI specification by an in-depth consideration of recent innovations in DL algorithms and their properties, which drive the need for specific fault tolerance features. We present an in-depth discussion on the suitability of different parallelism types (model, data and hybrid); a need (or lack thereof) for check-pointing of any critical data structures; and most importantly, consideration for several fault tolerance proposals (user-level fault mitigation (ULFM), Reinit) in MPI and their applicability to fault tolerant DL implementations. We leverage a distributed memory implementation of Caffe, currently available under the Machine Learning Toolkit for Extreme Scale (MaTEx). We implement our approaches by extending MaTEx-Caffe for using ULFM-based implementation. Our evaluation using the ImageNet dataset and AlexNet, and GoogLeNet neural network topologies demonstrates the effectiveness of the proposed fault tolerant DL implementation using OpenMPI based ULFM.

Deep Learning Reproducibility and Explainable AI (XAI)

“…Reproducibility is essential also in the context of fault tolerance, iterative refinement, debugging and optimization of adaptable models, especially for large scale and distributed workflow applications, like cloud computing platforms and Industry 4.0 [32]. The need for specific fault tolerance features for the properties of DL algorithms and their implementations has been already discussed elsewhere [2,36,31]. Reproducibility is also the basis for developing comparison criteria and metrics for the objective evaluation of model properties, like robustness and trustworthiness [33,19].…”

A Survey on Distributed Machine Learning

“…Although there are eorts to counteract some of this, production-ready solutions are lacking. Some of the described implementations allow for checkpointing to counteract this, but signicant eort is necessary to enable true fault-tolerance, as is described in Amatya et al [6]. It is also possible to reduce the probability of failure for each individual node, but this requires very specic hardware that is expensive and not generally available in commodity scale-out data centers or in the cloud.…”

A Study of Checkpointing in Large Scale Training of Deep Neural Networks

“…However, recent works show a growing interests. Early work by Vinay et al [4] addressed the requirements of MPI for designing fault tolerant DL applications, more specifically checkpoint-restart. Reagen et al [29] proposed a framework for quantifying the resilience DNNs.…”

“…While extensive research has been done to address most of these issues in many HPC workloads, not much has been done for DL workloads. For instance, not many frameworks are ideal for distributed training in supercomputers hence scaling DL training in distributed clusters often requires significant engineering effort [4]. To solve this problem, the DL community has created tools such as Horovod [5] that support distributed training on top of existing frameworks.…”