Matrix Engines for High Performance Computing: A Paragon of Performance or Grasping at Straws?

Domke, Jens; Vatai, Emil; Drozd, Aleksandr; Chen, Peng; Oyama, Yosuke; Zhang, Lingqi; Salaria, Shweta; Mukunoki, Daichi; Podobas, Artur; WahibT, Mohamed; Matsuoka, Satoshi

doi:10.1109/ipdps49936.2021.00114

Cited by 18 publications

(9 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although fast vectorization is powerful, matrix-multiplication performance could be improved further with specialized hardware units. This possibility is now realized with the introduction of what Domke et al have dubbed "matrix engines" 24 .…”

Section: Related Workmentioning

confidence: 99%

Fast matrix multiplication via compiler‐only layered data reorganization and intrinsic lowering

et al. 2023

View full text Add to dashboard Cite

The resurgence of machine learning has increased the demand for high-performance basic linear algebra subroutines (BLAS), which have long depended on libraries to achieve peak performance on commodity hardware. High-performance BLAS implementations rely on a layered approach that consists of tiling and packing layers -for data (re)organization -and micro kernels that perform the actual computations. The algorithm for the tiling and packing layers is target independent but is parameterized to the memory hierarchy and register-file size. The creation of high-performance micro kernels requires significant development effort to write tailored assembly code for each architecture. This hand optimization task is complicated by the recent introduction of matrix engines by IBM ® 's POWER10 ™ (Matrix Multiply Assist -MMA), Intel ® (Advanced Matrix eXtensions -AMX), and Arm ® (Matrix Extensions -ME) to deliver high-performance matrix operations. This paper presents a compiler-only alternative to the use of high-performance libraries by incorporating, to the best of our knowledge and for the first time, the automatic generation of the layered approach into LLVM, a production compiler. Modular design of the algorithm, such as the use of LLVM's matrix-multiply intrinsic for a clear interface between the tiling and packing layers and the micro kernel, makes it easy to retarget the code generation to multiple accelerators. The parameterization of the tiling and packing layers is demonstrated in the generation of code for the MMA unit on IBM's POWER10. This paper also describes an algorithm that lowers the matrix-multiply intrinsic to the MMA unit. The use of intrinsics enables a comprehensive performance study. In processors without hardware matrix engines, the tiling and packing delivers performance up to 22× (Intel) -for small matrices -and more than 6× (POWER9) -for large matrices -faster than PLuTo, a widely used polyhedral optimizer. The performance also approaches high-performance libraries and is only 34% slower than OpenBLAS and on-par with Eigen for large matrices. With MMA in POWER10 this solution is, for large matrices, over 2.6× faster the vector-extension solution, matches Eigen performance, and achieves up to 96% of BLAS peak performance.

show abstract

Section: Related Workmentioning

confidence: 99%

Fast matrix multiplication via compiler‐only layered data reorganization and intrinsic lowering

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Overall, many of the proxy applications listed above have been widely used for procurement decisions, vendor interactions (where full applications are too complex or classified), evaluation of programming models and testbeds, and supercomputing research all the way from transistor to full system scales (see various related scientific papers and presentations [14][15][16][17][18][19][20][21][22][23] ). However, the listed proxies are not the only ones available.…”

Section: State-of-the-art For Co-design Tool Chains and Procurement B...mentioning

confidence: 99%

Preparing for the Future -- Rethinking Proxy Apps

Matsuoka¹,

Domke²,

Wahib³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

A considerable amount of research and engineering went into designing proxy applications, which represent common high-performance computing workloads, to co-design and evaluate the current generation of supercomputers, e.g., RIKEN's Supercomputer Fugaku, ANL's Aurora, or ORNL's Frontier. This process was necessary to standardize the procurement while avoiding duplicated effort at each HPC center to develop their own benchmarks. Unfortunately, proxy applications force HPC centers and providers (vendors) into a an undesirable state of rigidity, in contrast to the fast-moving trends of current technology and future heterogeneity. To accommodate an extremelyheterogeneous future, we have to reconsider how to co-design supercomputers during the next decade, and avoid repeating the past mistakes. This position paper outlines the current state-of-the-art in system co-design, challenges encountered over the past years, and a proposed plan to move forward.

show abstract

“…Through the recent resurgence of hardware architecture for AI [48], most solutions are matrix-engine based [19] (Graphcore, Habana, Baidu Kunlun, Alibaba HanGuang, Huawei Ascend). Qualcomm's AI-100 seems to be based on VLIW DSPs.…”

Section: Related Workmentioning

confidence: 99%

The Mozart reuse exposed dataflow processor for AI and beyond

Sankaralingam

Nowatzki

Gangadhar

et al. 2022

Proceedings of the 49th Annual International Symposium on Computer Architecture

View full text Add to dashboard Cite

In this paper we introduce the Mozart Processor, which implements a new processing paradigm called Reuse Exposed Dataflow (RED). RED is a counterpart to existing execution models of Von-Neumann, SIMT, Dataflow, and FPGA. Dataflow and data reuse are the fundamental architecture primitives in RED, implemented with mechanisms for inter-worker communication and synchronization. The paper defines the processor architecture, the details of the microarchitecture, chip implementation, software stack development, and performance results. The architecture's goal is to achieve near-CPU like flexibility while having ASIC-like efficiency for a large-class of data-intensive workloads. An additional goal was software maturity -have large coverage of applications immediately, avoiding the need for a long-drawn hand-tuning software development phase. The architecture was defined with this software-maturity/compiler friendliness in mind. In short, the goal was to do to GPUs, what GPUs did to CPUs -i.e. be a better solution for a large range of workloads, while preserving flexibility and programmability. The chip was implemented with HBM and PCIe interfaces and taken to production on a 16nm TSMC FFC process. For ML inference tasks with batch-size=4, Mozart is integer factors better than state-of-theart GPUs even while being nearly 2 technology nodes behind. We conclude with a set of lessons learned, the unique challenges of a clean-slate architecture in a commercial setting, and pointers for uncovered research problems. CCS CONCEPTS• Computer systems organization → Data flow architectures;• Hardware → Hardware accelerators.

show abstract

Matrix Engines for High Performance Computing: A Paragon of Performance or Grasping at Straws?

Cited by 18 publications

References 31 publications

Fast matrix multiplication via compiler‐only layered data reorganization and intrinsic lowering

Fast matrix multiplication via compiler‐only layered data reorganization and intrinsic lowering

Preparing for the Future -- Rethinking Proxy Apps

The Mozart reuse exposed dataflow processor for AI and beyond

Contact Info

Product

Resources

About