Language support for dynamic, hierarchical data partitioning

TreichlerSean,; BauerMichael,; AikenAlex,

doi:10.1145/2544173.2509545

Cited by 11 publications

(4 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Legion [6,34] is a data-centric programming model and runtime for executing applications on distributed parallel systems. Legion maps each computation step to its data dependencies (i.e., the input of one step is the output of another).…”

Section: Related Workmentioning

confidence: 99%

Unity

Jones

Brim

Vallée

et al. 2017

Proceedings of the 7th International Workshop on Runtime and Operating Systems for Supercomputers ROSS 2017

View full text Add to dashboard Cite

This paper describes the vision for UNITY , a new high-performance computing focused data storage abstraction that places the entire memory hierarchy, including both traditionally separated memoryand file-based data storage, into one storage continuum. Through the use of a novel API and a set of services centered around a smart runtime system, UNITY is able to provide a number of valuable and interesting benefits. The unified storage space provides a scalable and resilient data environment that dynamically manages the mapping of data onto available resources based on multiple factors, including desired persistence and energy budget considerations. By eliminating the need for high-performance computing domain scientists to develop architecture-dependent optimizations for rapidly evolving data storage technologies, UNITY addresses both ease-of-use and performance.

show abstract

Section: Related Workmentioning

confidence: 99%

Unity

Jones

Brim

Vallée

et al. 2017

Proceedings of the 7th International Workshop on Runtime and Operating Systems for Supercomputers ROSS 2017

View full text Add to dashboard Cite

show abstract

“…Here we will just briefly review how we uniformly represent memory hierarchies in the framework, and how we have implemented the thread-to-cache affinities. [1] , [2] , [3] , [4] , [5] , [6] , [7]] , " size " : 524288 , " cacheLineSize " : 64 , " child " : { " siblings " : [[ ] , [1] , [2] , [3] , [4] , [5] , [6] , [7]] , " size " : 65536 , " cacheLineSize " : 64 , " child " : n u l l } } } } Listing 1: A NUMA node comprising two quad-core CPUs and 8 GBytes of RAM -4 Gbytes per CPU. Each CPU features a single L3 cache and four L1 and L2 caches -one per core.…”

Section: Implementation Detailsmentioning

confidence: 99%

“…Cache-guided optimizations in mainstream compilers consist essentially of loop transformations [1] directed at sequential loops, which, in the context of parallel computing, may only be applied to the internal execution of tasks. In fact, the data locality issue in parallel computing has been mostly addressed at language level, via linguistic constructions for the explicit programming of the memory hierarchy [2,3,4,5,6,7]. However, these place a heavy burden on the programmer, requiring in-depth knowledge of parallel programming and computer architecture.…”

Section: Introductionmentioning

confidence: 99%

Cache-conscious run-time decomposition of data parallel computations

Paulino

Delgado

2016

J Supercomput

View full text Add to dashboard Cite

Multi-core architectures feature an intricate hierarchy of cache memories, with multiple levels and sizes. To adequately decompose an application according to the traits of a particular memory hierarchy is a cumbersome task that may be rewarded with significant performance gains. The current state-of-the-art in memory hierarchyaware parallel computing delegates this endeavour on the programmer, demanding from him deep knowledge of both parallel programming and computer architecture. In this paper, we propose the shifting of these memory hierarchy-related concerns to the run-time system, which then takes on the responsibility of distributing the computation's data across the target memory hierarchy. We evaluate our approach from a performance perspective, comparing it against the common cache-neglectful data decomposition strategy.

show abstract

“…Programs written for Sequoia are composed of two parts: (a) an algorithmic representation of the computation using a C-like programming language that decomposes data structures and denes how to map the computation on them; and (b) a mapping of the algorithm to the specic system using a declarative language. Legion [94] is a generalization of Sequoia that exposes abstractions to declare the properties of program data. Programmers dynamically organize data into regions and dene how regions are accessed (i.e., access privileges) and computed.…”

Section: Languages For Multi-gpu Executionmentioning

confidence: 99%

On the programmability of multi-GPU computing systems

Rodríguez¹

View full text Add to dashboard Cite

Multi-GPU systems are widely used in High Performance Computing environments to accelerate scientific computations. This trend is expected to continue as integrated GPUs will be introduced to processors used in multi-socket servers and servers will pack a higher number of GPUs per node. GPUs are currently connected to the system through the PCI Express interconnect, which provides limited bandwidth (compared to the bandwidth of the memory in GPUs) and it often becomes a bottleneck for performance scalability. Current programming models present GPUs as isolated devices with their own memory, even if they share the host memory with the CPU. Programmers explicitly manage allocations in all GPU memories and use primitives to communicate data between GPUs. Furthermore, programmers are required to use mechanisms such as command queues and inter-GPU synchronization. This explicit model harms the maintainability of the code and introduces new sources for potential errors. The first proposal of this thesis is the HPE model. HPE builds a simple, consistent programming interface based on three major features. (1) All device address spaces are combined with the host address space to form a Unified Virtual Address Space. (2) Programs are provided with an Asymmetric Distributed Shared Memory system for all the GPUs in the system. It allows to allocate memory objects that can be accessed by any GPU or CPU. (3) Every CPU thread can request a data exchange between any two GPUs, through simple memory copy calls. Such a simple interface allows HPE to provide always the optimal implementation; eliminating the need for application code to handle different system topologies. Experimental results show improvements on real applications that range from 5% in compute-bound benchmarks to 2.6x in communication-bound benchmarks. HPE transparently implements sophisticated communication schemes that can deliver up to a 2.9x speedup in I/O device transfers. The second proposal of this thesis is a shared memory programming model that exploits the new GPU capabilities for remote memory accesses to remove the need for explicit communication between GPUs. This model turns a multi-GPU system into a shared memory system with NUMA characteristics. In order to validate the viability of the model we also perform an exhaustive performance analysis of remote memory accesses over PCIe. We show that the unique characteristics of the GPU execution model and memory hierarchy help to hide the costs of remote memory accesses. Results show that PCI Express 3.0 is able to hide the costs of up to a 10% of remote memory accesses depending on the access pattern, while caching of remote memory accesses can have a large performance impact on kernel performance. Finally, we introduce AMGE, a programming interface, compiler support and runtime system that automatically executes computations that are programmed for a single GPU across all the GPUs in the system. The programming interface provides a data type for multidimensional arrays that allows for robust, transparent distribution of arrays across all GPU memories. The compiler extracts the dimensionality information from the type of each array, and is able to determine the access pattern in each dimension of the array. The runtime system uses the compiler-provided information to automatically choose the best computation and data distribution configuration to minimize inter-GPU communication and memory footprint. This model effectively frees programmers from the task of decomposing and distributing computation and data to exploit several GPUs. AMGE achieves almost linear speedups for a wide range of dense computation benchmarks on a real 4-GPU system with an interconnect with moderate bandwidth. We show that irregular computations can also benefit from AMGE, too. Los sistemas multi-GPU son muy comúnmente utilizados en entornos de computación de altas prestaciones para acelerar cálculos científicos. Esta tendencia continuará con la introducción de GPUs integradas en los procesadores de los servidores procesador y con una mayor densidad de GPUs por nodo. Las GPUs actualmente se contectan al sistema a través de una interconexión PCI Express, que provee un ancho de banda reducido (comparado con las memorias de las GPUs) y habitualmente se convierte en el cuello de botella para escalar el rendimiento. Los modelos de programación actuales exponen las GPUs como dispositivos aislados con su propia memoria, incluso si comparten la memoria física con la CPU. Los programadores manejan diferentes reservas en todas las memorias de GPU y usan primitivas para comunicar datos entre GPUs. Además, los programadores deben utilizar mecanismos como colas de comandos y sincronicación entre GPUs. Este modelo explícito empeora la programabilidad del código e introduce nuevas fuentes de errores potenciales. La primera propuesta de esta tesis es el modelo HPE. HPE construye una interfaz de programaci ón consistente basada en tres características principales. (1) Todos los espacios de direcciones de los dispositivos son combinados para formar un espacio de direcciones unificado. (2) Los programas usan un sistema asimétrico distribuido de memoria compartida para todas las GPUs del sistema, que permite declarar objetos de memoria que pueden ser accedidos por cualquier GPU o CPU. (3) Cada hilo de ejecución de la CPU puede lanzar un intercambio de datos entre dos GPUs a través de simples llamadas de copia de memoria. Esta interfaz simplificada permite a HPE usar la implementaci ón óptima; sinque la aplicación contemple diferentes topologías de sistema. Los resultados experimentales muestran mejoras en aplicaciones reales que van desde un 5% en aplicaciones limitadas por el cómputo a 2.6x aplicaciones imitadas por la comunicación. HPE implementa sofisticados esquemas de transferencia para dispositivos de E/S que proporcionan mejoras de rendimiento de 2.9x. La segunda propuesta de esta tesis es un modelo de programación basado en memoria compartida que aprovecha las nuevas capacidades acceso remoto de memoria de las GPUs para eliminar la comunicación explícita entre memorias de GPU. Este modelo convierte un sistema multi-GPU en un sistema de memoria compartida con características NUMA. Para validar la viabilidad del modelo realizamos un anlásis exhaustivo del rendimiento los accessos de memoria remotos sobre PCIe. Los resultados muestran que PCI Express 3.0 elimina los costes de hasta un 10% de accesos remotos, dependiendo en el patrón de acceso, mientras que guardar los accesos remotos en memorias cache tiene un gran inpacto en el rendimiento de las computaciones. Finalmente, presentamos AMGE, una interfaz de programación con soporte de compilación y un sistema que ejecuta, de forma automática, computaciones programadas para una única GPU en todas las GPUs del sistema. La interfaz de programación proporciona un tipo de datos para arreglos multidimensionales que permite una distribuci ón transparente y robusta de los datos en todas las memorias de GPU. El compilador extrae la información sobre la dimensionalidad de cada arreglo y puede determinar el patrón de acceso en cada dimensión de forma individual. El sistema utiliza, en tiempo de ejecución, la información del compilador para elegir la mejor descomposición de la computación y los datos para minimizar la comunicación entre GPUs y el uso de memoria. AMGE consigue mejoras de rendimiento que crecen de forma lineal con el número de GPUs para un amplio abanico de computaciones densas en un sistema real con 4 GPUs. También mostramos que las computaciones con patrones irregulares también se pueden beneficiar de AMGE.

show abstract

Language support for dynamic, hierarchical data partitioning

Cited by 11 publications

References 37 publications

Unity

Unity

Cache-conscious run-time decomposition of data parallel computations

On the programmability of multi-GPU computing systems

Contact Info

Product

Resources

About