Scratchpad Sharing in GPUs

Abstract

GPGPU applications exploit on-chip scratchpad memory available in the Graphics Processing Units (GPUs) to improve performance. The amount of thread level parallelism (TLP) present in the GPU is limited by the number of resident threads, which in turn depends on the availability of scratchpad memory in its streaming multiprocessor (SM). Since the scratchpad memory is allocated at thread block granularity, part of the memory may remain unutilized. In this paper, we propose architectural and compiler optimizations to improve the scratchpad memory utilization. Our approach, called Scratchpad Sharing, addresses scratchpad under-utilization by launching additional thread blocks in each SM. These thread blocks use unutilized scratchpad memory and also share scratchpad memory with other resident blocks. To improve the performance of scratchpad sharing, we propose Owner Warp First (OWF) scheduling that schedules warps from the additional thread blocks effectively. The performance of this approach, however, is limited by the availability of the part of scratchpad memory that is shared among thread blocks.We propose compiler optimizations to improve the availability of shared scratchpad memory. We describe a scratchpad allocation scheme that helps in allocating scratchpad variables such that shared scratchpad is accessed for short duration. We introduce a new hardware instruction, relssp, that when executed, releases the shared scratchpad memory. Finally, we describe an analysis for optimal placement of relssp instructions such that shared scratchpad memory is released as early as possible, but only after its last use, along every execution path.We implemented the hardware changes required for scratchpad sharing approach and the new instruction (relssp) using the GPGPU-Sim simulator, and implemented the compiler optimizations in Ocelot framework. We evaluated the effectiveness of our approach on 19 kernels from 3 benchmarks suites: CUDA-SDK, GPGPU-Sim, and Rodinia. The kernels that under-utilize scratchpad memory show an average improvement of 19% and maximum improvement of 92.17% compared to the baseline approach, without affecting the performance of the kernels that do not waste scratchpad memory.

X-cache

“…Jenga [33] and Hotpads [34] organize the hierarchy of caches as a collection of SRAM banks. GPUs [16] and many cores use software-managed scratchpads. These approaches target CPUs or GPUs that role in address-translation and walking into the software.…”