Coordinated Batching and DVFS for DNN Inference on GPU Accelerators

Abstract

Deployment of real-time ML services on warehouse-scale infrastructures is on the increase. Therefore, decreasing latency and increasing throughput of deep neural network (DNN) inference applications that empower those services have attracted attention from both academia and industry. A common solution to address this challenge is leveraging hardware accelerators such as GPUs. To improve the inference throughput of DNNs deployed on GPU accelerators, two common approaches are employed: Batching and Multi-Tenancy. Our preliminary experiments show that the effect of these approaches on the throughput depends on the DNN architecture. Taking this observation into account, we design and implement DNNScaler which aims to maximize the throughput of interactive AI-powered services while meeting their latency requirements. DNNScaler first detects the suitable approach (Batching or Multi-Tenancy) that would be most beneficial for a DNN regarding throughput improvement. Then, it adjusts the control knob of the detected approach (batch size for Batching and number of co-located instances for Multi-Tenancy) to maintain the latency while increasing the throughput. Conducting an extensive set of experiments using well-known DNNs from a variety of domains, several popular datasets, and a cutting-edge GPU, the results indicate that DNNScaler can improve the throughput by up to 14x (218% on average) compared with the previously proposed approach, while meeting the latency requirements of the services.

Out of kernel tuning and optimizations for portable large-scale docking experiments on GPUs

“…When we focus on the application level, improving the computation efficiency by executing a batch of input together has been successfully applied in various fields [26,28,31]. This trend is even more common in machine learning for accelerating training and inference [15,18]. However, they target a more dynamic environment, considering the batch size as a tuning parameter [3].…”

Dependable DNN Accelerator for Safety-Critical Systems: A Review on the Aging Perspective

“…The mapping algorithm balances the positive error and the negative error of approximation to maximize the energy reduction while minimizing the overall approximation error. Finally, to tackle DVFS for power reduction, the forefront work of [151] introduces a new control knob based on the size of input batches fed to the DNN inference in the GPU. The authors first analyzed the effects of batch size on power and performance.…”

A Deep-Neural-Network-Based Decoding Scheme in Wireless Communication Systems

“…This makes it possible for DNN to be applied to communication systems without being limited by specific mathematical models. Moreover, DNN computations can be easily parallelized, which means they can take advantage of modern hardware accelerators such as GPUs to achieve faster training and inference speeds [8].…”