On-Device Learning Systems for Edge Intelligence: A Software and Hardware Synergy Perspective

Abstract

Modern machine learning (ML) applications are often deployed in the cloud environment to exploit the computational power of clusters. However, this in-cloud computing scheme cannot satisfy the demands of emerging edge intelligence scenarios, including providing personalized models, protecting user privacy, adapting to real-time tasks and saving resource cost. In order to conquer the limitations of conventional in-cloud computing, it comes the rise of on-device learning, which makes the endto-end ML procedure totally on user devices, without unnecessary involvement of the cloud. In spite of the promising advantages of on-device learning, implementing a high-performance on-device learning system still faces with many severe challenges, such as insufficient user training data, backward propagation blocking and limited peak processing speed. Observing the substantial improvement space in the implementation and acceleration of on-device learning systems, we intend to present a comprehensive analysis of the latest research progress and point out potential optimization directions from the system perspective. This survey presents a software and hardware synergy of on-device learning techniques, covering the scope of model-level neural network design, algorithm-level training optimization and hardware-level instruction acceleration. We hope this survey could bring fruitful discussions and inspire the researchers to further promote the field of edge intelligence.

ODSearch: A Fast and Resource Efficient On-device Information Retrieval for Mobile and Wearable Devices

“…Information storage is not typically an issue with cloud systems, but every query involves delays, both in the cloud and on-device. [15,57,74]. To reduce the delay, we analyzed several lossless text compression algorithms including, Run-length encoding, Shannon-Fano encoding, Arithmetic encoding, Huffman encoding, and LZW compression [61].…”

Aerial Computing: A New Computing Paradigm, Applications, and Challenges

“…Moreover, it is also necessary to develop on-device learning solutions to support self-learning aerial computing systems. For example, improved network architecture, training optimization, and hardware design were used to accelerate ondevice data training [159]. A streamlined slimming framework was developed and combined with a consecutive tensor layer to improve the training rates.…”

A Secure and Effective Energy-Aware Fixed-Point Quantization Scheme for Asynchronous Federated Learning

“…Fixed-point quantization could effectively reduce this high energy consumption and optimize the effect of neural network training [29]. There have been many recent studies focusing on fixed-point quantization [22,[30][31][32][33][34][35], which is summarized in Table 2. Proposed work Focuses [31] Integer-only quantization scheme 8-bit quantization, quantized inference framework, quantized training framework [32] INT8 training method (Octo) Quantization error, INT8 training [33] Relaxed Quantization (RQ) Network discretization, "Smooth" quantization procedure [34] Quantization-interval-learning (QIL) Quantization in low bit-width network, Trainable quantization interval [35] Data-free quantization method (DFQ) algorithm…”

“…Jacob B et al proposed a quantized inference framework, which could quantize both the weights and activations in the neural network [31]. Based on the first two works, a lightweight INT8 training method was proposed, in which both forward and backward stages were optimized by fixed point quantization [32]. The above researches deeply discuss the problems encountered when fixed-point quantization is applied to neural network training and give effective solutions.…”