Enabling pipeline parallelism in heterogeneous managed runtime environments via batch processing

Abstract

During the last decade, managed runtime systems have been constantly evolving to become capable of exploiting underlying hardware accelerators, such as GPUs and FPGAs. Regardless of the programming language and their corresponding runtime systems, the majority of the work has been focusing on the compiler front trying to tackle the challenging task of how to enable just-in-time compilation and execution of arbitrary code segments on various accelerators. Besides this challenging task, another important aspect that defines both functional correctness and performance of managed runtime systems is that of automatic memory management. Although automatic memory management improves productivity by abstracting away memory allocation and maintenance, it hinders the capability of using specific memory regions, such as pinned memory, in order to perform data transfer times between the CPU and hardware accelerators.In this paper, we introduce and evaluate a series of memory optimizations specifically tailored for heterogeneous managed runtime systems. In particular, we propose: (i) transparent and automatic "parallel batch processing" for overlapping data transfers and computation between the host and hardware accelerators in order to enable pipeline parallelism, and (ii) "off-heap pinned memory" in combination with parallel batch processing in order to increase the performance of data transfers without posing any on-heap overheads. These two techniques have been

Cross-Language Interoperability of Heterogeneous Code

“…Similarly to JVM, TornadoVM (any version prior to v0.14) allocates a memory space on each targeted hardware device that acts as a device memory heap [2]. That memory space is a replica of the heap memory space that is used by a Java application, and it is transferred on the device memory (i.e., DDR memory) in order to allow any heterogeneous code that executes on the device to access the data in the heap.…”

“…Although Java programmers that use TornadoVM are abstracted from the knowledge of heterogeneous programming models (e.g., OpenCL, CUDA, OneAPI), the heterogeneous code that is generated can be executed only within TornadoVM. This is due to numerous challenges, with the most severe one lying on the TornadoVM memory management system which allocates a memory space in device that enacts as a device memory heap [2]; similar to the JVM heap.…”