Programming for parallelism and locality with hierarchically tiled arrays

Abstract

Tiling has proven to be an effective mechanism to develop high performance implementations of algorithms. Tiling can be used to organize computations so that communication costs in parallel programs are reduced and locality in sequential codes or sequential components of parallel programs is enhanced.In this paper, a data type -Hierarchically Tiled Arrays or HTAs -that facilitates the direct manipulation of tiles is introduced. HTA operations are overloaded array operations. We argue that the implementation of HTAs in sequential OO languages transforms these languages into powerful tools for the development of high-performance parallel codes and codes with high degree of locality. To support this claim, we discuss our experiences with the implementation of HTAs for MATLAB and C++ and the rewriting of the NAS benchmarks and a few other programs into HTA-based parallel form.

Design and Use of htalib – A Library for Hierarchically Tiled Arrays

“…A Hierarchically Tiled Array (HTA) [7,4] is a recursive array data type where elements are either HTAs or scalars (at the bottom of the recursion). HTAs adopt tiling as a first class construct for array-based computations and empower programmers to control data distribution and the granularity of computation explicitly through the specification of tiling [2,8,9,16,18].…”

“…In earlier work, we introduced the concepts of programming with HTAs [4] and an early prototype based on MATLAB [7]. This paper describes htalib, a portable C++ library and framework for HTAs, and three application-driven extensions to the original HTA proposal [4] that facilitate the use of HTAs in certain application domains and broaden the potential application scope.…”

Multidimensional Blocking in UPC

“…Tiled data structures are used to enhance locality (and therefore performance) in a wide range of HPC applications [2]. Multiblocked arrays can help UPC programmers to better express these types of applications, allowing the language to fulfill its promise of allowing both high productivity and high performance.…”

“…Tiled and block distributions are useful for many linear algebra and scientific codes [2]. HPF-1 provided the ability to choose a data distribution independently in each dimesion if desired.…”

“…Beside HPF, several other languages, such as ZPL [3] and X10 [15] provide them as standard distributions supported by the language. In addition, libraries such as the Hierarichical Tiled Arrays library [2] provide tiled distributions for data decomposition. ScaLAPACK [6], a widely used parallel library provides a 2 dimensional block-cyclic distribution for matrices which allows the placement of blocks over a 2-dimensional processor grid.…”