Machine Learning Systems are Stuck in a Rut

Barham, Paul; Isard, Michael

doi:10.1145/3317550.3321441

Cited by 55 publications

(41 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the general structure of their monolithic hardware design is fixed, while Shir uses fine-grained multipurpose templates, that can be composed to implement the desired behaviour. Shir's approach is beneficial for the expressiveness, maintainability, and modularity, according to [4].…”

Section: Related Workmentioning

confidence: 99%

Memory-Aware Functional IR for Higher-Level Synthesis of Accelerators

Schlaak

Juang

Dubach

2022

ACM Trans. Archit. Code Optim.

View full text Add to dashboard Cite

Specialized accelerators deliver orders of a magnitude of higher performance than general-purpose processors. The ever-changing nature of modern workloads is pushing the adoption of Field Programmable Gate Arrays (FPGAs) as the substrate of choice. However, FPGAs are hard to program directly using Hardware Description Languages (HDLs). Even modern high-level HDLs, e.g., Spatial and Chisel, still require hardware expertise. This article adopts functional programming concepts to provide a hardware-agnostic higher-level programming abstraction. During synthesis, these abstractions are mechanically lowered into a functional Intermediate Representation (IR) that defines a specific hardware design point. This novel IR expresses different forms of parallelism and standard memory features such as asynchronous off-chip memories or synchronous on-chip buffers. Exposing such features at the IR level is essential for achieving high performance. The viability of this approach is demonstrated on two stencil computations and by exploring the optimization space of matrix-matrix multiplication. Starting from a high-level representation for these algorithms, our compiler produces low-level VHSIC Hardware Description Language (VHDL) code automatically. Several design points are evaluated on an Intel Arria 10 FPGA, demonstrating the ability of the IR to exploit different hardware features. This article also shows that the designs produced are competitive with highly tuned OpenCL implementations and outperform hardware-agnostic OpenCL code.

show abstract

Section: Related Workmentioning

confidence: 99%

Memory-Aware Functional IR for Higher-Level Synthesis of Accelerators

Schlaak

Juang

Dubach

2022

ACM Trans. Archit. Code Optim.

View full text Add to dashboard Cite

show abstract

“…The goal of our library is to provide a shape-safe interface for manipulating 𝑛-dimensional arrays (abbreviated ndarrays), where array shapes and indices are checked for errors at compile-time rather than at runtime. Shape and indexing errors in ndarrays is a widely acknowledged problem [Barham and Isard 2019;Rush 2019], and several solutions have already been proposed, notably in the form of libraries that rely on type-level programming [Chen 2017;Huang et al 2021]. Our library uses match types to provide a shape-safe NumPy-like interface.…”

Section: Case Study: Shape-safe Numpymentioning

confidence: 99%

Type-level programming with match types

Blanvillain

Brachthäuser

Kjaer

et al. 2022

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

Type-level programming is becoming more and more popular in the realm of functional programming. However, the combination of type-level programming and subtyping remains largely unexplored in practical programming languages. This paper presents match types , a type-level equivalent of pattern matching. Match types integrate seamlessly into programming languages with subtyping and, despite their simplicity, offer significant additional expressiveness. We formalize the feature of match types in a calculus based on System F sub and prove its soundness. We practically evaluate our system by implementing match types in the Scala 3 reference compiler, thus making type-level programming readily available to a broad audience of programmers.

show abstract

“…Though the toolchains are exceptionally feature-rich, there is evidence that some implementations have been highly-engineered for specific workloads, at the cost of general support for newer optimizations [4]. All current approaches, however, are limited by their inability to exploit NAS transformations.…”

Section: Related Workmentioning

confidence: 99%

Neural architecture search as program transformation exploration

Crowley

O’Boyle

2021

Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems

View full text Add to dashboard Cite

Improving the performance of deep neural networks (DNNs) is important to both the compiler and neural architecture search (NAS) communities. Compilers apply program transformations in order to exploit hardware parallelism and memory hierarchy. However, legality concerns mean they fail to exploit the natural robustness of neural networks. In contrast, NAS techniques mutate networks by operations such as the grouping or bottlenecking of convolutions, exploiting the resilience of DNNs. In this work, we express such neural architecture operations as program transformations whose legality depends on a notion of representational capacity. This allows them to be combined with existing transformations into a unified optimization framework. This unification allows us to express existing NAS operations as combinations of simpler transformations. Crucially, it allows us to generate and explore new tensor convolutions. We prototyped the combined framework in TVM and were able to find optimizations across different DNNs, that significantly reduce inference time -over 3× in the majority of cases. Furthermore, our scheme dramatically reduces NAS search time. Code is available at this https url. CCS CONCEPTS• Computing methodologies → Machine learning; • Software and its engineering → Compilers.

show abstract

Machine Learning Systems are Stuck in a Rut

Cited by 55 publications

References 7 publications

Memory-Aware Functional IR for Higher-Level Synthesis of Accelerators

Memory-Aware Functional IR for Higher-Level Synthesis of Accelerators

Type-level programming with match types

Neural architecture search as program transformation exploration

Contact Info

Product

Resources

About