Automatic array alignment in parallel Matlab scripts

Milosavljević, Igor; Jabri, Marwan A.

doi:10.1109/ipps.1999.760489

Cited by 10 publications

(5 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prior work in this space proposes static, compile-time techniques for analysis. The first set of techniques focuses on partitioning [29]; the second on data co-location [37,49,55]. Prior work also has examined nested loops with affine array subscript patterns, using different structures (vector [29], matrix [60] or reference [32]) to model memory access patterns or polyhedral model [45] to perform localization analysis.…”

Section: Related Workmentioning

confidence: 99%

Unifying Data, Model and Hybrid Parallelism in Deep Learning via Tensor Tiling

Wang,

Huang,

2018

Preprint

View full text Add to dashboard Cite

Deep learning systems have become vital tools across many fields, but the increasing model sizes mean that training must be accelerated to maintain such systems' utility. Current systems like TENSORFLOW and MXNET focus on one specific parallelization strategy, data parallelism, which requires large training batch sizes in order to scale. We cast the problem of finding the best parallelization strategy as the problem of finding the best tiling to partition tensors with the least overall communication. We propose an algorithm that can find the optimal tiling. Our resulting parallelization solution is a hybrid of data parallelism and model parallelism. We build the SOYBEAN system that performs automatic parallelization. SOYBEAN automatically transforms a serial dataflow graph captured by an existing deep learning system frontend into a parallel dataflow graph based on the optimal tiling it has found. Our evaluations show that SOYBEAN is 1.5×−4× faster than pure data parallelism for AlexNet and VGG. We present this automatic tiling in a new system, SOYBEAN, that can act as a backend for TENSORFLOW, MXNET, and others.

show abstract

Section: Related Workmentioning

confidence: 99%

Unifying Data, Model and Hybrid Parallelism in Deep Learning via Tensor Tiling

Wang,

Huang,

2018

Preprint

View full text Add to dashboard Cite

show abstract

“…Several of these are similar to Matlab D in the sense that they compile Matlab to C or Fortran, either generating assertions for a parallel Fortran compiler or calls to parallel numerical libraries. Included in this class are Falcon [12], Otter [13], CONLAB [14], and ParAL [15]. While these projects obtain improved performance by compiling to low-level languages, they do not take advantage of the performance possible from being able to automatically specialize libraries for additional distributions.…”

Section: Related Workmentioning

confidence: 99%

Compiling Parallel MATLAB for General Distributions using Telescoping Languages

Fletcher

McCosh

Jin

et al. 2007

2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07

View full text Add to dashboard Cite

Matlab is one of the most popular computer languages for technical and scienti c programming. However, until recently, it has been limited to running on uniprocessors. One strategy for overcoming this limitation is to introduce global distributed arrays, with those arrays distributed across the processors of a parallel machine. In this paper, we describe the compilation technology we have designed for Matlab D, a distributed-array extension of Matlab. Our approach is distinguished by a two-phase compilation technology with support for a rich collection of data distributions. By precompiling array operations and communication steps into Fortran plus MPI, the time to compile an application using those operations is signi cantly reduced. This paper includes preliminary results that demonstrate that this approach can dramatically improve performance, scaling well to at least 32 processors .

show abstract

“…Software that makes use of this approach: Otter [40], RT-Express [28], ParAL [36], FALCON [18], CONLAB [20], MATCH [7], and Menhir [14] These projects compile MATLAB scripts into an executable, sometimes translating the scripts into a compiled language as an intermediate step. Some projects, e.g., [40] and [20], link the compiled code with parallel numerical libraries, while some software, e.g., [18], generates code that is already parallel.…”

Section: Matlab Compilersmentioning

confidence: 99%

Parallel MATLAB: Doing it Right

2005

View full text Add to dashboard Cite

MATLAB is one of the most widely used mathematical computing environments in technical computing. It is an interactive environment that provides high-performance computational routines and an easy-to-use, C-like scripting language. It started out as an in-1) Memory model: Distributed memory was the dominant model for parallel computers, and for linear algebra applications, scatter/gather of the matrix took too long to make parallel computation worthwhile. 2) Granularity: For typical use, MATLAB spends most of its time in the parser, interpreter and graphics routines, where any parallelism is difficult to find. Also, to handle embarrassingly parallel applications, which only requires a collection of results at the end, MATLAB would require fundamental changes in its architecture. 3) Business situation: There were not enough customers with parallel computers to support the development. It has been nine years since the article was written, and we have seen tremendous changes in the computing world. These changes have invalidated the arguments that there should not be a parallel MATLAB. 1) Memory model: As modern scientific and engineering problems grow in complexity, the computation time and memory requirements skyrocket. The increase in processor speed and the amount of memory that can fit in a single machine could not catch up with the pace of computation requirements. Very often, current scientific problems simply do not fit into the memory of a single machine, making parallel computation a necessity. This has always been true throughout the history of computing-problems just do not fit into memory. But we have hit the point where a lot of interesting and practical problems from different areas do not fit into memory of a single machine. This has made parallel computing a necessity.

show abstract

Automatic array alignment in parallel Matlab scripts

Cited by 10 publications

References 9 publications

Unifying Data, Model and Hybrid Parallelism in Deep Learning via Tensor Tiling

Unifying Data, Model and Hybrid Parallelism in Deep Learning via Tensor Tiling

Compiling Parallel MATLAB for General Distributions using Telescoping Languages

Parallel MATLAB: Doing it Right

Contact Info

Product

Resources

About