Research and optimization of intra-node communication operator performance based on domestic heterogeneous platforms

Abstract

With the increasing demand for computing power in machine learning tasks, the training of deep neural network models has been pushed to multi-GPU training or even larger scale distributed training. However, the acceleration effect and scalability of model training are largely limited by the communication efficiency between GPUs. In order to improve the communication efficiency of domestic GPU accelerator, this paper studies and analyzes the communication performance of Allreduce operator widely used in deep learning tasks. Based on a data compression algorithm and multi-stream parallel technology, this paper ports and optimizes the Allreduce operator on domestic heterogeneous platform. The experimental data results show that compared with RCCL, the ported optimized version of Allreduce operator achieves different degrees of performance improvement such as 30% to 90% under different data sizes. This paper implements the transplantation and optimization of Allreduce operator on the domestic accelerated heterogeneous platform, and achieves good acceleration results, which provides support for the efficient communication of domestic GPU accelerators and the ecological construction of domestic heterogeneous platforms.