Real-Time Sound Source Localization for Low-Power IoT Devices Based on Multi-Stream CNN

Abstract

Voice-activated artificial intelligence (AI) technology has advanced rapidly and is being adopted in various devices such as smart speakers and display products, which enable users to multitask without touching the devices. However, most devices equipped with cameras and displays lack mobility; therefore, users cannot avoid touching them for face-to-face interactions, which contradicts the voice-activated AI philosophy. In this paper, we propose a deep neural network-based real-time sound source localization (SSL) model for low-power internet of things (IoT) devices based on microphone arrays and present a prototype implemented on actual IoT devices. The proposed SSL model delivers multi-channel acoustic data to parallel convolutional neural network layers in the form of multiple streams to capture the unique delay patterns for the low-, mid-, and high-frequency ranges, and estimates the fine and coarse location of voices. The model adapted in this study achieved an accuracy of 91.41% on fine location estimation and a direction of arrival error of 7.43° on noisy data. It achieved a processing time of 7.811 ms per 40 ms samples on the Raspberry Pi 4B. The proposed model can be applied to a camera-based humanoid robot that mimics the manner in which humans react to trigger voices in crowded environments.

Machine Learning Assists IoT Localization: A Review of Current Challenges and Future Trends

“…The authors investigated voice-activated AI technology and proposed a deep-neural-network-based real-time sound source localization (SSL) model for low-power IoT devices. The authors used multichannel acoustic data to parallelize convolutional neural network layers in the form of multiple streams in order to capture unique delay patterns in the low-, mid-, and high-frequency ranges and estimate the fine and coarse location of voices [ 58 ].…”

“… Classification of the reviewed literature, with emphasis on the learning algorithms used [ 28 , 29 , 31 , 32 , 33 , 34 , 37 , 38 , 39 , 40 , 41 , 43 , 44 , 46 , 47 , 49 , 50 , 51 , 52 , 56 , 57 , 58 , 59 , 60 , 61 , 62 ]. …”

Energy Neutral Urban Noise Monitoring and Classification With LoRaWAN Based IoT

“…As an alternative to signal processing based DoA estimation methods, end-to-end ML models eliminate feature extraction step and enable lowcomputation DoA prediction. However, the best offerings of the current literature focus on platforms with significantly higher power consumption than battery-powered nodes used in low-cost IoT-based noise monitoring systems [45], [46].…”

Gaussian Process Regression for Single-Channel Sound Source Localization System Based on Homomorphic Deconvolution

“…The flying drone localizes the sound source from ambient noise and frequent movement [ 19 ]. The low-power device with a microphone array derives the AoA for the sound source in the application of various human interaction [ 20 ]. The complex environment from the indoor condition is challenged for the SSL system by Machhamer [ 21 ] and Zhang [ 22 ].…”