Batch Size Influence on Performance of Graphic and Tensor Processing Units During Training and Inference Phases

Abstract

The impact of the maximally possible batch size (for the better runtime) on performance of graphic processing units (GPU) and tensor processing units (TPU) during training and inference phases is investigated. The numerous runs of the selected deep neural network (DNN) were performed on the standard MNIST and Fashion-MNIST datasets. The significant speedup was obtained even for extremely low-scale usage of Google TPUv2 units (8 cores only) in comparison to the quite powerful GPU NVIDIA Tesla K80 card with the speedup up to 10x for training stage (without taking into account the overheads) and speedup up to 2x for prediction stage (with and without taking into account overheads). The precise speedup values depend on the utilization level of TPUv2 units and increase with the increase of the data volume under processing, but for the datasets used in this work (MNIST and Fashion-MNIST with images of sizes 28x28) the speedup was observed for batch sizes >512 images for training phase and >40 000 images for prediction phase. It should be noted that these results were obtained without detriment to the prediction accuracy and loss that were equal for both GPU and TPU runs up to the 3rd significant digit for MNIST dataset, and up to the 2nd significant digit for Fashion-MNIST dataset.

Linear-Time Self Attention with Codeword Histogram for Efficient Recommendation

“…The computation of Eq. ( 7) therefore operates on different sizes of tensors, which is sub-optimal for efficient batching in GPU and TPU [20]. For batching purpose, we perform the attention over all codewords in each codebook, fixing the "context size" of the attention to 𝑊 :…”

A Distributed ML Framework for Service Deployment in the 5G-based Automotive Vertical

“…It offers cloud services specialized for ML (Amazon EC2 P3 instances) and is equipped with an NVIDIA Tesla V100 Graphics Processing Unit (GPU). GPU computational capacity values for ML can be found in [18], where we select an average value of 6000 training samples/sec. Finally, the computational tasks for the cloud server include training (in the CML case) and model parameter aggregation (FML, EML cases).…”

“…For the former we select an average value of 40,000 training samples/sec, assuming a Data Center is equipped with a Tensor Processing Unit (TPU). For the latter (model parameter aggregation), no reference values can be found in the literature, thus we rely on an empirical approach; we measure the average capacity for training and aggregation tasks in our personal computer (PC) setup (i.e., 6250 training samples/sec and 1.56 model aggregations/sec respectively) and compare against the training capacity reference value of 40,000 training samples/sec that was selected, according to [18]. Assuming a linear relation, the average cloud aggregation capacity is calculated as 10 model aggregations/sec.…”

On the Resource Consumption of Distributed ML

“…It offers cloud services specialized for ML and is equipped with an NVIDIA Tesla V100 Graphics Processing Unit (GPU). Thus, an edge node's computational capacity equals that of a GPU's computational capacity, whose values for ML tasks can be found in [19], from where we select an average value of =6000 training samples/sec.…”