A Novel Mixed Precision Distributed TPU GAN for Accelerated Learning Curve

Abstract

Deep neural networks are gaining importance and popularity in applications and services. Due to the enormous number of learnable parameters and datasets, the training of neural networks is computationally costly. Parallel and distributed computation-based strategies are used to accelerate this training process. Generative Adversarial Networks (GAN) are a recent technological achievement in deep learning. These generative models are computationally expensive because a GAN consists of two neural networks and trains on enormous datasets. Typically, a GAN is trained on a single server. Conventional deep learning accelerator designs are challenged by the unique properties of GAN, like the enormous computation stages with non-traditional convolution layers. This work addresses the issue of distributing GANs so that they can train on datasets distributed over many TPUs (Tensor Processing Unit). Distributed learning training accelerates the learning process and decreases computation time. In this paper, the Generative Adversarial Network is accelerated using the distributed multi-core TPU in distributed data-parallel synchronous model. For adequate acceleration of the GAN network, the data parallel SGD (Stochastic Gradient Descent) model is implemented in multi-core TPU using distributed TensorFlow with mixed precision, bfloat16, and XLA (Accelerated Linear Algebra). The study was conducted on the MNIST dataset for varying batch sizes from 64 to 512 for 30 epochs in distributed SGD in TPU v3 with 128 × 128 systolic array. An extensive batch technique is implemented in bfloat16 to decrease the storage cost and speed up floating-point computations. The accelerated learning curve for the generator and discriminator network is obtained. The training time was reduced by 79% by varying the batch size from 64 to 512 in multi-core TPU.

Computationally Efficient Neural Rendering for Generator Adversarial Networks Using a Multi-GPU Cluster in a Cloud Environment

“…The impact of the multinode on spark and GPU was examined for medical use cases [35], [38]. The impact of multi-node TPU on GAN for double precision was developed but lacked deployment and did not address the current bottleneck issues in TPU [34]. The proposed model is superior to the existing works since it uses multi-GPU GAN implementation addressing the bottleneck issues, ensures the deployment and continuous retraining of the model, and makes it usable for real-time applications.…”

DPro-SM – A distributed framework for proactive straggler mitigation using LSTM

“…Utilizing high-performance hardware isn't the only option; one potential alternative is to parallelize and provide DNN training operations on many nodes instead. Under these circumstances, the amount of work that each node contributes to the calculation is minimal at best [ [3] , [4] , [5] , [6] , [7] ]. Despite this, the communication delay is a critical obstacle in distributed training due to the frequent communication requirements for delivering vast volumes of data across multiple computing nodes.…”