Towards Ubiquitous Intelligent Computing: Heterogeneous Distributed Deep Neural Networks

Abstract

For the pursuit of ubiquitous computing, distributed computing systems containing the cloud, edge devices, and Internet-of-Things devices are highly demanded. However, existing distributed frameworks do not tailor for the fast development of Deep Neural Network (DNN), which is the key technique behind many intelligent applications nowadays. Based on prior exploration on distributed deep neural networks (DDNN), we propose Heterogeneous Distributed Deep Neural Network (HDDNN) over the distributed hierarchy, targeting at ubiquitous intelligent computing. While being able to support basic functionalities of DNNs, our framework is optimized for various types of heterogeneity, including heterogeneous computing nodes, heterogeneous neural networks, and heterogeneous system tasks. Besides, our framework features parallel computing, privacy protection and robustness, with other consideration for the combination of heterogeneous distributed system and DNN. Extensive experiments demonstrate that our framework is capable of utilizing hierarchical distributed system better for DNN and tailoring DNN for real-world distributed system properly, which is with low response time, high performance, and better user experience.Manuscript received xx, xx; revised xx, xx.

Pervasive AI for IoT Applications: A Survey on Resource-Efficient Distributed Artificial Intelligence

“…Hierarchical distribution can also be combined with compressing strategies to reduce the size of the data to be transmitted and accordingly minimize the communication delay and the time of the entire inference, such as using the encoding techniques as done in [221]. Authors in [222], [223] also opted for hierarchical offloading, while focusing primarily on fault-tolerance of the shared data. Particularly, authors in [222] considered two fault-tolerance methods, namely reassigning and monitoring, where the first one consists of assigning all layers tasks at least once, and then the unfinished tasks are reassigned to all participants regardless of their current state.…”

“…Authors in [222], [223] also opted for hierarchical offloading, while focusing primarily on fault-tolerance of the shared data. Particularly, authors in [222] considered two fault-tolerance methods, namely reassigning and monitoring, where the first one consists of assigning all layers tasks at least once, and then the unfinished tasks are reassigned to all participants regardless of their current state. This method, is generating a considerable communication and latency overhead related to allocating redundant tasks, particularly to devices with limitedcapacities.…”

RL-PDNN: Reinforcement Learning for Privacy-Aware Distributed Neural Networks in IoT Systems

“…The second partition strategy is the hierarchical splitting where the offloading is accomplished across the cloud, edge servers, and mobile devices. HDDNN [22] is one of the hierarchical partitioning approaches, where authors tested the performance of the distributed system with heterogeneous nodes' capacities, heterogeneous DNN networks, and heterogeneous tasks.…”

Pervasive AI for IoT Applications: Resource-efficient Distributed Artificial Intelligence

“…Hierarchical distribution can also be combined with compressing strategies to reduce the size of the data to be transmitted and accordingly minimize the communication delay and the time of the entire inference, such as using the encoding technique as done in [166]. Authors in [167], [168] also opted for hierarchical offloading, while focusing primarily on fault-tolerance of the shared data. Particularly, authors in [167] considered two fault-tolerance methods, namely reassigning and monitoring, where the first one consists of assigning all layers tasks at least once, and then the unfinished tasks are reassigned to all participants regardless of their current state.…”

“…Authors in [167], [168] also opted for hierarchical offloading, while focusing primarily on fault-tolerance of the shared data. Particularly, authors in [167] considered two fault-tolerance methods, namely reassigning and monitoring, where the first one consists of assigning all layers tasks at least once, and then the unfinished tasks are reassigned to all participants regardless of their current state. This method, is generating a considerable communication and latency overhead related to allocating redundant tasks, particularly to devices with limitedcapacities.…”