Industrial Cyber-Physical Systems-Based Cloud IoT Edge for Federated Heterogeneous Distillation

Abstract

Deep convoloutional networks have been widely deployed in modern cyber-physical systems performing different visual classification tasks. As the fog and edge devices have different computing capacity and perform different sub-tasks, models trained for one device may not be deployable on another. Knowledge distillation technique can effectively compress well trained convolutional neural networks (CNN) into lightweight models suitable to different devices. However, due to privacy issue and transmission cost, manually annotated data for training the deep learning models are usually gradually collected and archived in different sites. Simply training a model on powerful cloud servers and compressing them for particular edge devices failed to use the distributed data stored at different sites. This offline training approach is also inefficient to deal with new data collected from the edge devices. To overcome these obstacles, we propose the heterogeneous brain storming (HBS) method for object recognition tasks in real-world IoT scenarios. Our method enables flexible bidirectional federated learning of heterogeneous models trained on distributed datasets with a new "brain storming" mechanism and optimizable temperature parameters. In our comparison experiments, this heterogeneous brain storming method outperformed multiple state-of-the-art single-model compression methods, as well as the newest multinetwork knowledge distillation methods with both homogeneous and heterogeneous classifiers. The ablation experiment results proved that the trainable temperature parameter into the conventional knowledge distillation loss can effectively ease the learning process of student networks in different methods. To the best of

Knowledge distillation in deep learning and its applications

“…The idea of HAR based on spatio-temporal features from IoT devices like a cup, a toothbrush and a fork was presented in Lopez Medina et al (2019) . Knowledge distillation was also used for training a small model for image classification which will help IoT-based security systems to detect intrusion ( Wang et al (2020) ).…”

Advance Warning Methodologies for COVID-19 Using Chest X-Ray Images

“…C ORONAVIRUS disease 2019 (COVID- 19) caused by severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2) has declared as pandemic by the World Health Organization (WHO) on the 11th of March, 2020. Although, infected patients tend to have mild and unspecific symptoms [1] such as fever, myalgia or fatigue, and cough, the disease affects seriously people in high-risk groups especially the elderly.…”

“…Nevertheless, there are existing studies addressing these drawbacks, for example, the method in [18] is able to reduce the volume of the training data by 10% using a semi-supervised based approach. Moreover, for the latter, the method in [19] proposes a knowledge distillation approach to transfer CNN models to source-limited devices where only light-weight models can be fit and suitable for the inference. Next, the need for such lightweight conversions is discussed in [20] for IoT devices.…”

Heuristic Edge Server Placement in Industrial Internet of Things and Cellular Networks

“…With mobile and IoT devices permeating in every field of life, services such as robotics, automation, assembly and production, machine intelligence, and virtual reality are becoming more and more widespread. These services, whether targeting IIoT in industry 4.0 [2]- [3] or infotainment and emergency services in cellular networks [4], [5]- [7] require high computing power and are sensitive to delays in communication networks [8].…”