Toward Collaborative Inferencing of Deep Neural Networks on Internet-of-Things Devices

Abstract

With recent advancements in deep neural networks (DNNs), we are able to solve traditionally challenging problems. Since DNNs are compute intensive, consumers, to deploy a service, need to rely on expensive and scarce compute resources in the cloud. This approach, in addition to its dependability on high-quality network infrastructure and data centers, raises new privacy concerns. These challenges may limit DNN-based applications, so many researchers have tried optimize DNNs for local and in-edge execution. However, inadequate power and computing resources of edge devices along with small number of requests limits current optimizations applicability, such as batch processing. In this paper, we propose an approach that utilizes aggregated existing computing power of Internet of Things (IoT) devices surrounding an environment by creating a collaborative network. In this approach, IoT devices cooperate to conduct single-batch inferencing in real time. While exploiting several new model-parallelism methods and their distribution characteristics, our approach enhances the collaborative network by creating a balanced and distributed processing pipeline. We have illustrated our work using many Raspberry Pis with studying DNN models such as AlexNet, VGG16, Xception, and C3D.

Investigating Mobile Edge-Cloud Trade-Offs of Object Detection with YOLO

“…applying different approaches for model-parallelism. The results of Hadidi et al [2019] show that the collaborative network is enhanced by creating a distributed processing pipeline. Differently from these works, we are interested in the partitioning of DNN models between edge and cloud.…”

“…The results show that their proposed method is able to reduce the network communication costs without harming accuracy. A similar approach is proposed by Hadidi et al [2019], which aggregates existing computing power of edge devices in an local network environment by creating a collaborative network. In this scenario, edge devices cooperate to make inferences by…”

Adaptive parallel execution of deep neural networks on heterogeneous edge devices

“…Collaborative perception [18] pipelines the computation by partitioning a DNN model and distributing the partitioned blocks to multiple edge devices. Follow-up work in [17] introduces model parallelism methods for both dense and convolution layers in order to address the memory overhead due to model replication. DeepThings [51] reduces the communication cost by fusing the early convolution layers and parallelizing these layers in multiple devices.…”

“…In response to the aforementioned concerns, efforts have been made to push machine learning inference from the cloud to the edge. Edge processing has the benefit of keeping data closer to its source to provide real-time responses while protecting the privacy of the end-user [17,29]. Unfortunately, edge computing for machine learning workloads faces challenges of its own.…”

“…Some prior work [17,33] explored parallelizing DNN inference on edge devices. Their approach, however, relies on layer-granularity parallelism that result in large amounts of intermediate feature maps that are transferred across devices.…”