MNN: A Universal and Efficient Inference Engine

Jiang, Xiaotang; Wang, Huan; Chen, Yiliu; Wu, Ziqi; Wang, Lichuan; Zou, Bin; Yang, Yan; Cui, Zongyang; Cai, Yu-Dong; Yu, Tianhang; Lv, Chengfei; Wu, Zhihua

doi:10.48550/arxiv.2002.12418

Cited by 15 publications

(19 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As neural network models become increasingly commonplace in production, there have been a number of different open and closed source inference frameworks from both academia and industry. Some notable examples include OpenVINO for CPUs, TensorRT for Nvidia GPUs and MNN for mobile devices [15,25,31]. Almost all deep learning inference engines consist of two stages: preparation and execution.…”

Section: Deep Learning Inference Enginesmentioning

confidence: 99%

“…OpenVINO and TensorRT rely on Intel oneDNN (formerly MKL-DNN) and Nvidia cuDNN respectively [15,31]. MNN relies on a semi-automated search technique to generate the kernels from a pre-defined number of optimization strategies [25]. TVM takes it a step further and performs compilation and autotuning for each kernel [2].…”

Section: Deep Learning Inference Enginesmentioning

confidence: 99%

“…This section describes the architecture of the SparseDNN sparse deep learning inference system. The overall architecture, illustrated in Figure 1, is similar to other inference systems such as OpenVINO, TensorRT and MNN [15,25,31].…”

Section: Sparsednn Architecturementioning

confidence: 99%

“…Memory buffers that store input activations for earlier layers of the network can be repurposed and rewritten in later layers. This is a simple optimization that is often done to minimize the memory footprint of the inference engine on edge devices [25]. However, on sparse networks, it also has a big impact on inference performance.…”

Section: Activation Buffermentioning

confidence: 99%

“…We show that through both network-level and kernel-level optimizations, we can significantly speed up the execution of unstructured sparse neural networks on modern datacenter CPUs where most deep learning inference takes place today [13]. The design of SparseDNN is inspired by other state-of-the-art inference systems for dense neural networks such as TensorRT, OpenVINO and MNN [15,25,31], as well as performance engineering research targeting sparse matrix multiplications and convolutions in Intel Libxsmm and SkimCaffe [14,32]. SparseDNN leverages both novel networklevel optimizations such as weight permutation and kernel-level optimizations through a sparse kernel generator which achieves significantly superior performance than Libxsmm and SkimCaffe [14,32].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

SparseDNN: Fast Sparse Deep Learning Inference on CPUs

Wang¹

2021

Preprint

View full text Add to dashboard Cite

The last few years have seen gigantic leaps in algorithms and systems to support efficient deep learning inference. Pruning and quantization algorithms can now consistently compress neural networks by an order of magnitude. For a compressed neural network, a multitude of inference frameworks have been designed to maximize the performance of the target hardware. While we find mature support for quantized neural networks in production frameworks such as OpenVINO and MNN, support for pruned sparse neural networks is still lacking. To tackle this challenge, we present SparseDNN, a sparse deep learning inference engine targeting CPUs. We present both kernel-level optimizations with a sparse code generator to accelerate sparse operators and novel network-level optimizations catering to sparse networks. We show that our sparse code generator can achieve significant speedups over state-of-the-art sparse and dense libraries. On end-to-end benchmarks such as Huggingface pruneBERT, SparseDNN achieves up to 5x throughput improvement over dense inference with state-of-the-art OpenVINO.

show abstract

Section: Deep Learning Inference Enginesmentioning

confidence: 99%

Section: Deep Learning Inference Enginesmentioning

confidence: 99%

Section: Sparsednn Architecturementioning

confidence: 99%

Section: Activation Buffermentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

SparseDNN: Fast Sparse Deep Learning Inference on CPUs

Wang¹

2021

Preprint

View full text Add to dashboard Cite

show abstract

Efficient grouping for keypoint detection

Sidnev

Krasikova²,

Kazakov

2021

2020 25th International Conference on Pattern Recognition (ICPR)

View full text Add to dashboard Cite

Fusion-Catalyzed Pruning for Optimizing Deep Learning on Intelligent Edge Devices

Wang

et al. 2020

IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst.

View full text Add to dashboard Cite

The increasing computational cost of deep neural network models limits the applicability of intelligent applications on resource-constrained edge devices. While a number of neural network pruning methods have been proposed to compress the models, prevailing approaches focus only on parametric operators (e.g., convolution), which may miss optimization opportunities. In this paper, we present a novel fusion-catalyzed pruning approach, called FUPRUNER, which simultaneously optimizes the parametric and non-parametric operators for accelerating neural networks. We introduce an aggressive fusion method to equivalently transform a model, which extends the optimization space of pruning and enables non-parametric operators to be pruned in a similar manner as parametric operators, and a dynamic filter pruning method is applied to decrease the computational cost of models while retaining the accuracy requirement. Moreover, FUPRUNER provides configurable optimization options for controlling fusion and pruning, allowing much more flexible performance-accuracy trade-offs to be made. Evaluation with state-of-the-art residual neural networks on five representative intelligent edge platforms, Jetson TX2, Jetson Nano, Edge TPU, NCS, and NCS2, demonstrates the effectiveness of our approach, which can accelerate the inference of models on CIFAR-10 and ImageNet datasets.

show abstract

MNN: A Universal and Efficient Inference Engine

Cited by 15 publications

References 0 publications

SparseDNN: Fast Sparse Deep Learning Inference on CPUs

SparseDNN: Fast Sparse Deep Learning Inference on CPUs

Efficient grouping for keypoint detection

Fusion-Catalyzed Pruning for Optimizing Deep Learning on Intelligent Edge Devices

Contact Info

Product

Resources

About