Communication and computation optimization of concurrent kernels using kernel coalesce on a GPU

Abstract

General purpose computation on graphics processing unit (GPU) is rapidly entering into various scientific and engineering fields. Many applications are being ported onto GPUs for better performance. Various optimizations, frameworks, and tools are being developed for effective programming of GPU. As part of communication and computation optimizations for GPUs, this paper proposes and implements an optimization method called as kernel coalesce that further enhances GPU performance and also optimizes CPU to GPU communication time. With kernel coalesce methods, proposed in this paper, the kernel launch overheads are reduced by coalescing the concurrent kernels and data transfers are reduced incase of intermediate data generated and used among kernels. Computation optimization on a device (GPU) is performed by optimizing the number of blocks and threads launched by tuning it to the architecture. Block level kernel coalesce method resulted in prominent performance improvement on a device without the support for concurrent kernels. Thread level kernel coalesce method is better than block level kernel coalesce method when the design of a grid structure (i.e., number of blocks and threads) is not optimal to the device architecture that leads to underutilization of the device resources. Both the methods perform similar when the number of threads per block is approximately the same in different kernels, and the total number of threads across blocks fills the streaming multiprocessor (SM) capacity of the device. Thread multi-clock cycle coalesce method can be chosen if the programmer wants to coalesce more than two concurrent kernels that together or individually exceed the thread capacity of the device. If the kernels have light weight thread computations, multi clock cycle kernel coalesce method gives better performance than thread and block level kernel coalesce methods. If the kernels to be coalesced are a combination of compute intensive and memory intensive kernels, warp interleaving gives higher device occupancy and improves the performance. Multi clock cycle kernel coalesce method for micro-benchmark1 considered in this paper resulted in 10-40% and 80-92% improvement compared with separate kernel launch, without and with shared input and intermediate data among the kernels, respectively, on a Fermi architecture device, that is, GTX 470. A nearest neighbor (NN) kernel from Rodinia benchmark is coalesced to itself using thread level kernel coalesce method and warp interleaving giving 131.9% and 152.3% improvement compared with separate kernel launch and 39.5% and 36.8% improvement compared with block level kernel coalesce method, respectively. 48 B. NEELIMA, G. R. M. REDDY AND P. S. RAGHAVENDRA models such as compute unified device architecture (CUDA), Open specification for Compute Language. The data parallel applications are ported onto GPU, and GPU can give higher performances than CPU for such applications. The resource allocation on GPU is defined at the grid level by the programmer. The programs have usually a g...

Predicting an Optimal Sparse Matrix Format for SpMV Computation on GPU

“…This tool not only combines all the memory transfers of multiple kernels to be sent to GPU but also optimizes the performance if the kernels use the shared data. Bayyapu et al [15] also emphasized the importance of CPU to GPU communication time and proposed a tool named kernel coalesce to reduce the number of kernel calls by merging concurrent kernels and optimized the GPU execution time by increasing the device utilization and by reducing the number of memory accesses for shared data among kernels.…”

A GPU Framework for Sparse Matrix Vector Multiplication

“…A Composite format that is a combination of ELL and COO (Co-ordinate format), named hybrid(HYB) is proposed by Bell and Garland [8] and a combination of ELL and CSR (Compressed Sparse Row) formats is proposed by Anirudh et al [9] and Kiran and Kishore [10]. Further, CPU to GPU communication time importance and optimizations were given by various researchers ( [13]- [17]). …”

The Design and Analysis of Producer-Consumer Model Based on Multi-Mutual Exclusive Random Buffer Models

“…Otherwise, it will not design robust programs. Literature [5] studies the concurrency problem from the perspective of network programs, pointing out that the concurrency problem is an intrinsically complex problem, which makes traditional programming methods encounter many difficulties in the development of high-quality (concurrent) programs, and seriously affects the development efficiency. The impact of concurrency problems on the complexity of (concurrent) program development can be likened to the "software concurrency crisis" caused by multi-core processors.…”