Memory optimization at Edge for Distributed Convolution Neural Network

Abstract

Internet of Things (IoT) edge intelligence has emerged by optimizing the deep learning (DL) models deployed on resource‐constraint devices for quick decision‐making. In addition, edge intelligence reduces network overload and latency by bringing intelligent analytics closer to the source. On the other hand, DL models need a lot of computing resources. As a result, they have high computational workloads and memory footprint, making it impractical to deploy and execute on IoT edge devices with limited capabilities. In addition, existing layer‐based partitioning methods generate many intermediate results, resulting in a huge memory footprint. In this article, we propose a framework to provide a comprehensive solution that enables the deployment of convolutional neural networks (CNNs) onto distributed IoT devices for faster inference and reduced memory footprint. This framework considers a pretrained YOLOv2 model, and a weight pruning technique is applied to the pre‐trained model to reduce the number of non‐contributing parameters. We use the fused layer partitioning method to vertically partition the fused layers of the CNN and then distribute the partition among the edge devices to process the input. In our experiment, we have considered multiple Raspberry Pi as edge devices. Raspberry Pi with a neural computing stick is a gateway device to combine the results from various edge devices and get the final output. Our proposed model achieved inference latency of 5 to ∼$$ \sim $$7 seconds for 3prefix×3$$ 3\times 3 $$ to 5prefix×5$$ 5\times 5 $$ fused layer partitioning for five devices with a 9% improvement in memory footprint.

Deep Reinforcement Learning-Empowered Cost-Effective Federated Video Surveillance Management Framework

“…We have summarized the state of the art (SOTA) in energy-efficient and DL-based video surveillance studies in Table 1 for the convenience of the reader. These approaches, as highlighted in [18][19][20][21][22], have achieved significant improvements in energy efficiency by leveraging edge computing and optimization mechanisms. This has reduced network bandwidth and response time in IoT-based smart video surveillance systems, enabling effective object detection and analysis of abnormal behavior.…”

“…Naveen et al [22] Saving GPU resources Suggested reducing the number of non-contributing parameters.…”

Cognitive Video Surveillance Management in Hierarchical Edge Computing System with Long Short-Term Memory Model

“…For instance, refs. [4][5][6][7][8] made significant achievements in energy efficiency through edge computing and optimizing mechanisms, resulting in a reduced network bandwidth and response time in IoT-based smart video surveillance systems for effective object detection and abnormal behavior analysis. Moreover, ref.…”

Compact optimized deep learning model for edge: a review

“…Pruning [11]- [15] removes redundant or unnecessary connections in a neural network. Pruning involves removing individual weights or removing entire neurons or layers.…”

“…Naveen et al [14], [15] 2021 For distributing the workload to each IoT edge device, the pre-trained model is pruned to lower the model parameter, and then fused tail partitioning is employed.…”