Enhanced Robot Speech Recognition Using Biomimetic Binaural Sound Source Localization

Abstract

Inspired by the behavior of humans talking in noisy environments, we propose an embodied embedded cognition approach to improve automatic speech recognition (ASR) systems for robots in challenging environments, such as with ego noise, using binaural sound source localization (SSL). The approach is verified by measuring the impact of SSL with a humanoid robot head on the performance of an ASR system. More specifically, a robot orients itself toward the angle where the signal-to-noise ratio (SNR) of speech is maximized for one microphone before doing an ASR task. First, a spiking neural network inspired by the midbrain auditory system based on our previous work is applied to calculate the sound signal angle. Then, a feedforward neural network is used to handle high levels of ego noise and reverberation in the signal. Finally, the sound signal is fed into an ASR system. For ASR, we use a system developed by our group and compare its performance with and without the support from SSL. We test our SSL and ASR systems on two humanoid platforms with different structural and material properties. With our approach we halve the sentence error rate with respect to the common downmixing of both channels. Surprisingly, the ASR performance is more than two times better when the angle between the humanoid head and the sound source allows sound waves to be reflected most intensely from the pinna to the ear microphone, rather than when sound waves arrive perpendicularly to the membrane.

What Can Computational Models Learn From Human Selective Attention? A Review From an Audiovisual Unimodal and Crossmodal Perspective

“…The aims of acoustic models for the Cocktail Party problem are: identifying multiple speakers and disentangling each speech stream from noisy background. Numerous classical acoustic models are data-driven and based on algorithms of signal processing (Dávila-Chacón et al, 2018). Those models are robust and with good accuracy but lack the prior knowledge, biological plausibility and rely on the large datasets.…”

Automatic Speech Recognition: Systematic Literature Review

“…At the same time, MAR models are widely used in forecasting. Within the same scope in [57], a neural network can be used to calculate the audio signal's angle. A forward-bound neural network is then used to deal with the noise.…”

“…Furthermore, background noise can cause telephone channel distortions; suitable system performance in the presence of background noise requires high-quality microphone manufacturing [53]. In addition, to apply the approaches that use beamforming for speech segregation, the number of microphones has to be larger than the number of sound sources [57].…”

“…Therefore, any locative property changes of the sound during the observation of received signals were caused by the change in the acoustic channel, which resulted in decreasing the performance and affecting the outcomes. In [57], the authors proposed an embedded cognition method to improve ASR for robots, using microphone arrays to locate the speech sources. They then separated the speech signals from background noise.…”

What can computational models learn from human selective attention? A review from an audiovisual crossmodal perspective

“…The aims of acoustic models for the Cocktail Party problem are: identifying multiple speakers and disentangling each speech stream from noisy background. Numerous classical acoustic models are data-driven and based on algorithms of signal processing (Dá vila-Chacó n et al, 2018). Those models are robust and with good accuracy but lack the prior knowledge, biological plausibility and rely on the large datasets.…”