Efficient Federated Learning for AIoT Applications Using Knowledge Distillation

Liu, Tian; Xia, Jun; Ling, Zhiwei; Fu, Xin; Yu, Shui; Chen, Mingsong

doi:10.1109/jiot.2022.3229374

Cited by 14 publications

(2 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another optimization approach is knowledge distillation, which transfers knowledge from a complex to a simpler model and has been used successfully for many models and applications [22], [23]. These model optimizations are useful to enable the use of ML in IoT devices; however, in most cases, the energy per inference still limits the number of inferences that can be made.…”

Section: A Machine Learning Optimizationsmentioning

confidence: 99%

Adaptive Resolution Inference (ARI): Energy-Efficient Machine Learning for Internet of Things

Wang,

Reviriego,

Niknia

et al. 2024

IEEE Internet Things J.

View full text Add to dashboard Cite

The implementation of Machine Learning (ML) in Internet of Things (IoT) devices poses significant operational challenges due to limited energy and computation resources. In recent years, significant efforts have been made to implement simplified ML models that can achieve reasonable performance while reducing computation and energy, for example by pruning weights in neural networks, or using reduced precision for the parameters and arithmetic operations. However, this type of approach is limited by the performance of the ML implementation, i.e., by the loss for example in accuracy due to the model simplification.In this paper, we present Adaptive Resolution Inference (ARI), a novel approach that enables to evaluate new tradeoffs between energy dissipation and model performance in ML implementations. The main principle of the proposed approach is to run inferences with reduced precision (quantization) and use the margin over the decision threshold to determine if either the result is reliable, or the inference must run with the full model. The rationale is that quantization only introduces small deviations in the inference scores, such that if the scores have a sufficient margin over the decision threshold, it is very unlikely that the full model would have a different result. Therefore, we can run the quantized model first, and only when the scores do not have a sufficient margin, the full model is run. This enables most inferences to run with the reduced precision model and only a small fraction requires the full model, so significantly reducing computation and energy while not affecting model performance. The proposed ARI approach is presented, analyzed in detail, and evaluated using different datasets both for floating-point and stochastic computing implementations. The results show that ARI can significantly reduce the energy for inference in different configurations with savings between 40% and 85%.

show abstract

Section: A Machine Learning Optimizationsmentioning

confidence: 99%

Adaptive Resolution Inference (ARI): Energy-Efficient Machine Learning for Internet of Things

Wang,

Reviriego,

Niknia

et al. 2024

IEEE Internet Things J.

View full text Add to dashboard Cite

show abstract

“…На рисунке 1 представлена данная архитектура. В статье [7] продемонстрирован подход, который использует Knowledge Distillation (KD). В этом подходе устройства обучают локальные модели на своих данных и отправляют на сервер не только градиенты моделей, но и мягкие цели (soft targets), которые представляют собой вероятности предсказаний для каждого класса.…”

Section: централизованная архитектураunclassified

Федеративное обучение для IoT и AIoT: применения, проблемы и перспективы

Eleev

2024

News of KBSC RAS

View full text Add to dashboard Cite

Научно-образовательный центр Кабардино-Балкарского научного центра Российской академии наук 360010, Россия, г. Нальчик, ул. Балкарова, 2Аннотация. В статье рассматривается концепция федеративного обучения (FL) -распределенного совместного подхода к искусственному интеллекту (AI), который позволяет обучать AI на распределенных IoT устройствах без необходимости обмена данными. Подходы и методы реализации FL для AIoT устройств были классифицированы по трем типам архитектуры федеративного обучения для организации взаимодействия между участниками обучения: централизованная, децентрализованная и гибридная. Рассмотрены подходы, основанные на различных технологиях, таких как Knowledge Distillation, блокчейн, беспроводные сети типа Mesh, Hybrid-IoT, DHA-FL. Для каждой рассмотренной технологии обозначены основные преимущества, проблемы и вызовы. В заключение сделаны выводы о перспективах развития FL для IoT и AIoT.

show abstract