Automatic generation of high-performance quantized machine learning kernels

Cowan, Meghan; Moreau, Thierry; Chen, Tianqi; Bornholt, James; Ceze, Luís

doi:10.1145/3368826.3377912

Cited by 31 publications

(22 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…a user-specified cost model that assigns a cost to each operation. Such a problem lies at the heart of many applications [5,[17][18][19]. Later, we supply the generated synthesis problems to a second synthesizer.…”

Section: Impact Of Algorithms 1 and 2 (Rq2)mentioning

confidence: 99%

“…Its ability to create a program that implements a specification function and consists of some given bit vector operations has recently propelled research in computer programming. For example, program synthesizers craft instruction selection rules in compilers [4], superoptimize code [18], generate code for unusual architectures [17], optimize machine learning kernels [5], or enumerate rewrite rules for SMT solvers [16]. As it is often not a priori known which and how many operations to use for a synthesized program, some applications formulate multiple synthesis problems that differ in the used operations [4] or in the length of the program [18].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Approximate Bit Dependency Analysis to Identify Program Synthesis Problems as Infeasible

Kamp

Philippsen

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Bit-vector-based program synthesis is an important building block of state-of-the-art techniques in computer programming. Some of these techniques do not only rely on a synthesizer's ability to return an appropriate program if it exists but also require a synthesizer to detect if there is no such program at all in the entire search space (i.e., the problem is infeasible), which is a computationally demanding task.In this paper, we propose an approach to quickly identify some synthesis problems as infeasible. We observe that a specification function encodes dependencies between input and output bits that a correct program must satisfy. To exploit this fact, we present approximate analyses of essential bits and use them in two novel algorithms to check if a synthesis problem is infeasible. Our experiments show that adding our technique to applications of bit vector synthesis can save up to 33% of their time.

show abstract

Section: Impact Of Algorithms 1 and 2 (Rq2)mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Approximate Bit Dependency Analysis to Identify Program Synthesis Problems as Infeasible

Kamp

Philippsen

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

Section: Related Workmentioning

confidence: 99%

“…Cowan et al [14] discusses a complementary, automated approach for implementing quantized inference that relies on the scheduling phase of a compiler and program synthesis techniques. Specifically, [14] takes as input a quantized inference kernel which we believe can be replaced by a SumMerge-based kernel. The result would be an optimized version of SumMerge that takes into account bitplane scheduling and other complementary optimizations discussed in [14].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

SumMerge

Prabhakar

Kuhar

Agrawal

et al. 2021

Proceedings of the ACM International Conference on Supercomputing

View full text Add to dashboard Cite

Deep Neural Network (DNN) inference efficiency is a key concern across the myriad of domains now relying on Deep Learning. A recent promising direction to speed-up inference is to exploit weight repetition. The key observation is that due to DNN quantization schemes-which attempt to reduce DNN storage requirements by reducing the number of bits needed to represent each weight-the same weight is bound to repeat many times within and across filters. This enables a weight-repetition aware inference kernel to factorize and memoize out common sub-computations, reducing arithmetic per inference while still maintaining the compression benefits of quantization. Yet, significant challenges remain. For instance, weight repetition introduces significant irregularity in the inference operation and hence (up to this point) has required custom hardware accelerators to derive net benefit. This paper proposes SumMerge: a new algorithm and set of implementation techniques to make weight repetition practical on general-purpose devices such as CPUs. The key idea is to formulate inference as traversing a sequence of data-flow graphs with weight-dependent structure. We develop an offline heuristic to select a data-flow graph structure that minimizes arithmetic operations per inference (given trained weight values) and use an efficient online procedure to traverse each data-flow graph and compute the inference result given DNN inputs. We implement the above as an optimized C++ routine that runs on a commercial multicore processor with vector extensions and evaluate performance relative to Intel's optimized library oneDNN and the prior-art weight repetition algorithm (AGR). When applied on top of six different quantization schemes, SumMerge achieves a speedup of between

show abstract

Quantune: Post-training quantization of convolutional neural networks using extreme gradient boosting for fast deployment

Lee

Kwon

et al. 2022

Future Generation Computer Systems

View full text Add to dashboard Cite

Automatic generation of high-performance quantized machine learning kernels

Cited by 31 publications

References 16 publications

Approximate Bit Dependency Analysis to Identify Program Synthesis Problems as Infeasible

Approximate Bit Dependency Analysis to Identify Program Synthesis Problems as Infeasible

SumMerge

Quantune: Post-training quantization of convolutional neural networks using extreme gradient boosting for fast deployment

Contact Info

Product

Resources

About