Best Practices for the Deployment of Edge Inference: The Conclusions to Start Designing

Flamis, Georgios; Kalapothas, Stavros; Kitsos, Paris

doi:10.3390/electronics10161912

Cited by 6 publications

(2 citation statements)

References 88 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The larger the DL model the more parameters it will have, and consequently the more memory space (in RAM) required to host the model. Model size or memory footprint is computed having 'MB' as their unit of measurement [243,244,245,246,247]. For a specific image classification problem, if MobileNet V2 with 3.54 million parameters is selected, it will have 14 MB as model size whereas if InceptionV4 with 42.74 million parameters is selected for the same problem, it will have a 163 MB model size requirement [242].…”

Section: H Memory Footprint/ Model Sizementioning

confidence: 99%

Enabling All In-Edge Deep Learning: A Literature Review

et al. 2023

View full text Add to dashboard Cite

In recent years, deep learning (DL) models have demonstrated remarkable achievements on non-trivial tasks such as speech recognition, image processing, and natural language understanding. One of the significant contributors to its success is the proliferation of end devices that acted as a catalyst to provide data for data-hungry DL models. However, computing DL training and inference is the main challenge. Usually, central cloud servers are used for the computation, but it opens up other significant challenges, such as high latency, increased communication costs, and privacy concerns. To mitigate these drawbacks, considerable efforts have been made to push the processing of DL models to edge servers (a mesh of computing devices near end devices). Moreover, the confluence point of DL and edge has given rise to edge intelligence (EI). International Electrotechnical Commission (IEC) defines EI as the concept where the data is acquired, stored, and processed utilizing edge computing with DL and advanced networking capabilities. Broadly, EI has six levels of categories based on where the training and inference of DL take place, e.g., cloud server, edge server and end devices. This survey paper focuses primarily on the fifth level of EI, called all in-edge level, where DL training and inference (deployment) are performed solely by edge servers. All in-edge is suitable when the end devices have low computing resources, e.g., Internet-of-Things, and other requirements such as latency and communication cost are important such as in mission-critical applications (e.g., health care). Besides, 5G/6G networks are envisioned to use all in-edge. Firstly, this paper presents all in-edge computing architectures, including centralized, decentralized, and distributed. Secondly, this paper presents enabling technologies, such as model parallelism, data parallelism, and split learning, which facilitates DL training and deployment at edge servers. Thirdly, model adaptation techniques based on model compression and conditional computation are described because the standard cloud-based DL deployment cannot be directly applied to all in-edge due to its limited computational resources. Fourthly, this paper discusses eleven key performance metrics to evaluate the performance of DL at all in-edge efficiently. Finally, several open research challenges in the area of all in-edge are presented. INDEX TERMS Artificial intelligence, all in-edge, deep learning, distributed systems, decentralized systems, edge intelligence I. INTRODUCTION T HE global community is increasingly becoming a datadriven environment in which end devices are generating vast quantities of data outside of the traditional data centers. International Telecommunication Union anticipates that global internet traffic per month will reach 607 Exabytes (EB) in 2025 and 5016 EB in 2030 [1]. This enormous amount of data has a positive impact on artificial intelligence (AI) applications. In particular, deep learning (DL) rely on the availability of large quantities of data for its d...

show abstract

Section: H Memory Footprint/ Model Sizementioning

confidence: 99%

Enabling All In-Edge Deep Learning: A Literature Review

et al. 2023

View full text Add to dashboard Cite

show abstract

“…As an EDGE server usually has limited infrastructure resources, it becomes challenging to host a DNN model because of the computational requirements (the bigger the network, the more parameters it will have and each extra parameter increases the memory requirement (in RAM)). Model size or memory footprint is computed having 'MB' as their unit of measurement [35,61,139,167,260]. For the image classification problem, if MobileNet V2 with 3.54 million parameters is selected it will have 14 MB as model size whereas if InceptionV4 with 42.74 million parameters is selected for the same problem it will have a 163 MB model size requirement [133].…”

Section: Memory Footprint/ Model Sizementioning

confidence: 99%

Enabling Deep Learning for All-in EDGE paradigm

Joshi¹,

Hasanuzzaman²,

Thapa³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep Learning-based models have been widely investigated, and they have demonstrated significant performance on non-trivial tasks such as speech recognition, image processing, and natural language understanding. However, this is at the cost of substantial data requirements. Considering the widespread proliferation of edge devices (e.g., Internet of Things devices) over the last decade, Deep Learning in the edge paradigm, such as device-cloud integrated platforms, is required to leverage its superior performance. Moreover, it is suitable from the data requirements perspective in the edge paradigm because the proliferation of edge devices has resulted in an explosion in the volume of generated and collected data. However, there are difficulties due to other requirements such as high computation, high latency, and high bandwidth caused by Deep Learning applications in real-world scenarios. In this regard, this survey paper investigates Deep Learning at the edge, its architecture, enabling technologies, and model adaption techniques, where edge servers and edge devices participate in deep learning training and inference. For simplicity, we call this paradigm the All-in EDGE paradigm. Besides, this paper presents the key performance metrics for Deep Learning at the All-in EDGE paradigm to evaluate various deep learning techniques and choose a suitable design. Moreover, various open challenges arising from the deployment of Deep Learning at the All-in EDGE paradigm are identified and discussed.

show abstract