DeepX: A Software Accelerator for Low-Power Deep Learning Inference on Mobile Devices

Abstract

Breakthroughs from the field of deep learning are radically changing how sensor data are interpreted to extract the high-level information needed by mobile apps. It is critical that the gains in inference accuracy that deep models afford become embedded in future generations of mobile apps. In this work, we present the design and implementation of DeepX, a software accelerator for deep learning execution. DeepX significantly lowers the device resources (viz. memory, computation, energy) required by deep learning that currently act as a severe bottleneck to mobile adoption. The foundation of DeepX is a pair of resource control algorithms, designed for the inference stage of deep learning, that: (1) decompose monolithic deep model network architectures into unit-blocks of various types, that are then more efficiently executed by heterogeneous local device processors (e.g., GPUs, CPUs); and (2), perform principled resource scaling that adjusts the architecture of deep models to shape the overhead each unit-blocks introduces. Experiments show, DeepX can allow even large-scale deep learning models to execute efficiently on modern mobile processors and significantly outperform existing solutions, such as cloud-based offloading.

A collaborative CPU‐GPU approach for deep learning on mobile devices

“…First, these research works target different computing devices, such as DSP, 5,32 GPU, 24 and LPU. 24 Several of these attempts take advantage of linear algebra-based optimization, such as singular value decomposition 33 and Tucker decomposition, 34 to reduce the complexity of convolution computations. 24 Several of these attempts take advantage of linear algebra-based optimization, such as singular value decomposition 33 and Tucker decomposition, 34 to reduce the complexity of convolution computations.…”

“…First, these research works target different computing devices, such as DSP, 5,32 GPU, 24 and LPU. 33 Second, these efforts aim to leverage several programming models on mobile devices, such as OpenCL, 34 Vulkan, 34 CUDA 33 (only available on compatible devices), and RenderScript. 24 Several of these attempts take advantage of linear algebra-based optimization, such as singular value decomposition 33 and Tucker decomposition, 34 to reduce the complexity of convolution computations.…”

RSTensorFlow

“…For example, DeepX [18] accelerates the deep learning inference on mobile devices by using the DSP, GPU and using runtime layer compression to decompose the deep model across available hardware resources. However, in their paper results, DeepX [18] used the GPU only on the Nvidia Tegra K1 Soc and relied on using DSP on the more popular Snapdragon Qualcomm SoC. Also, DeepX is not available for the public developers to use and does not integrate within popular deep learning frameworks.…”

“…Therefore, our can be used to accelerate other models than convolution neural networks. Finally, our system can be used to run models trained with TensorFlow out of the box without any model conversion or preparation as needed by [20], and [18]. …”

Optimizing convolutional neural networks on embedded platforms with OpenCL

“…We are also aiming to enable powerful CNN compression techniques [13,16] in the CNN optimization workflow and expose all their optimization parameters. Indeed, while optimizing the building blocks of CNNs is important, it is even more important to ensure that no unnecessary computation and data movement takes place.…”