Memory- and Communication-Aware Model Compression for Distributed Deep Learning Inference on IoT

Abstract

Model compression has emerged as an important area of research for deploying deep learning models on Internet-of-Things (IoT). However, for extremely memory-constrained scenarios, even the compressed models cannot fit within the memory of a single device and, as a result, must be distributed across multiple devices. This leads to a distributed inference paradigm in which memory and communication costs represent a major bottleneck. Yet, existing model compression techniques are not communication-aware. Therefore, we propose Network of Neural Networks (NoNN), a new distributed IoT learning paradigm that compresses a large pretrained 'teacher' deep network into several disjoint and highly-compressed 'student' modules, without loss of accuracy. Moreover, we propose a network science-based knowledge partitioning algorithm for the teacher model, and then train individual students on the resulting disjoint partitions. Extensive experimentation on five image classification datasets, for user-defined memory/performance budgets, show that NoNN achieves higher accuracy than several baselines and similar accuracy as the teacher model, while using minimal communication among students. Finally, as a case study, we deploy the proposed model for CIFAR-10 dataset on edge devices and demonstrate significant improvements in memory footprint (up to 24×), performance (up to 12×), and energy per node (up to 14×) compared to the large teacher model. We further show that for distributed inference on multiple edge devices, our proposed NoNN model results in up to 33× reduction in total latency w.r.t. a state-of-the-art model compression baseline.

EdgeAl: A Vision for Deep Learning in the IoT Era

“…With ever-more devices available on the edge, this network of devices must be exploited to improve model accuracy without increasing the communication latency. Towards this, we describe our recent work on Network-of-Neural Networks (NoNN) [1] for memoryand communication-aware model compression.…”

“…These patterns of activations reveal how knowledge learned by the teacher network is distributed at the final convolution layer. Therefore, we first use these patterns to create a filter activation network [1] which represents how knowledge is distributed across multiple filters (see Fig. 3(b)).…”

“…3(b)). We then partition this network into disjoint subsets via community detection [10], a network partitioning technique (see [1] for details).…”

“…However, due to their enormous computational complexity, deploying such models on constrained devices has emerged as a critical bottleneck for large-scale adoption of intelligence at the IoT edge. It has been estimated that the number of connected IoT-devices will reach one trillion across various market segments by 2035 1 ; this provides us a unique opportunity for integrating widespread intelligence in edge devices. Such an exponential growth in IoT-devices necessitates new breakthroughs in Artificial Intelligence research that can more effectively deploy learning at the edge and, therefore, truly exploit the setup of trillions of IoT-devices.…”

“…Finally, since IoT naturally implies a network of connected devices, it automatically opens the door to a new class of problems -communication-aware model compression [1]. For instance, many smart home/cities applications can have numerous connected IoT-sensors with, say, 500KB total memory per node.…”