Neural Network Distillation on IoT Platforms for Sound Event Detection

Abstract

In most classification tasks, wide and deep neural networks perform and generalize better than their smaller counterparts, in particular when they are exposed to large and heterogeneous training sets. However, in the emerging field of Internet of Things memory footprint and energy budget pose severe limits on the size and complexity of the neural models that can be implemented on embedded devices. The Student-Teacher approach is an attractive strategy to distill knowledge from a large network into smaller ones, that can fit on low-energy lowcomplexity embedded IoT platforms. In this paper, we consider the outdoor sound event detection task as a use case. Building upon the VGGish network, we investigate different distillation strategies to substantially reduce the classifier's size and computational cost with minimal performance losses. Experiments on the UrbanSound8K dataset show that extreme compression factors (up to 4.2 • 10 −4 for parameters and 1.2 • 10 −3 for operations with respect to VGGish) can be achieved, limiting the accuracy degradation from 75% to 70%. Finally, we compare different embedded platforms to analyze the trade-off between available resources and achievable accuracy.

PhiNets: A Scalable Backbone for Low-power AI at the Edge

“…On the other side, with the advent of deep learning techniques, machine learning algorithms' size grows exponentially, thanks to the improvements in processor speeds and the availability of large training data. However, embedded systems cannot sustain the resource requirements of standard deep learning techniques, adequate for GP-GPUs [6,14,33].…”

“…TinyML has been applied to several diferent classes of problems and devices, such as audio processing and sound event detection [6] [38], biosignals processing [21], gesture recognition [43] and general time series data [15]. Among the several application domains where to explore this novel trend, computer vision is one of the optimisation since it accounts for the most considerable computational cost in network inference.…”

PhiNets: a scalable backbone for low-power AI at the edge

“…On the other side, with the advent of deep learning techniques, machine learning algorithms' size grows exponentially, thanks to the improvements in processor speeds and the availability of large training data. However, embedded systems cannot sustain the resource requirements of standard deep learning techniques, adequate for GP-GPUs [4,11,25].…”

Compact Recurrent Neural Networks for Acoustic Event Detection on Low-Energy Low-Complexity Platforms

“…Going from state-of-the-art neural models to actual implementation on an IoT device involves multiple stages, as depicted in Fig 1 . In our previous publication [14], we presented a KD approach to compress a SED classifier composed of the publicly available VGGish feature extractor [15] and arXiv:2001.10876v1 [eess.AS] 29 Jan 2020 a recurrent classifier. Differently from common applications of KD, aimed at improving performance or at achieving limited reductions of the model dimensions, we obtained very high compression factors, reducing the network size from approximately 70 million parameters to nearly 20 thousand.…”

“…In this paper, we focus in particular on i) a preliminary analysis of the computational and memory requirements to understand what kind of models can be afforded by a given class of microcontrollers; ii) the quantization of the network parameters and the activations, presenting two different strategies to select for each layer the best fixed-point representation; iii) an implementation of the reduced network on a microcontroller with resources typical of an IoT endnode, building upon the network reduction strategies presented in [14] and iv) the evaluation of the accuracy of the actual implementation. In addition, we present an improvement of the KD approach presented in [14]. Distillation is performed in two stages where adaptation of the VGGish pre-trained feature extraction to the in-domain data is separated from the actual parameter distillation, leading to a further improvement of the classification accuracy.…”