Hybrid Neural Networks for On-Device Directional Hearing

Abstract

On-device directional hearing requires audio source separation from a given direction while achieving stringent human-imperceptible latency requirements. While neural nets can achieve significantly better performance than traditional beamformers, all existing models fall short of supporting low-latency causal inference on computationally-constrained wearables. We present DeepBeam, a hybrid model that combines traditional beamformers with a custom lightweight neural net. The former reduces the computational burden of the latter and also improves its generalizability, while the latter is designed to further reduce the memory and computational overhead to enable real-time and low-latency operations. Our evaluation shows comparable performance to state-of-the-art causal inference models on synthetic data while achieving a 5x reduction of model size, 4x reduction of computation per second, 5x reduction in processing time and generalizing better to real hardware data. Further, our real-time hybrid model runs in 8 ms on mobile CPUs designed for low-power wearable devices and achieves an end-to-end latency of 17.5 ms.

Semantic Hearing: Programming Acoustic Scenes with Binaural Hearables

“…To ensure that the audio played through the headset is synced with the user's visual senses, we need this end-to-end latency to be less than 20-50 ms [24,59,67]. To achieve this, we need to reduce the buffer duration, the look-ahead duration and the processing time.…”

“…3) Real-time operation requires processing each acoustic block within the duration of the block itself. This means that it should take less than 10 ms to process a 10 ms buffer [67]. This can be challenging since neural networks are not known for their lightweight computation.…”

“…Our inspiration for the name 'semantic hearing' is directional hearing which is the ability to hear sounds from a specific direction[10,16,67]. Similarly, semantic hearing is the ability to hear the sounds that are specified by some semantic descriptions, such as sound classes.…”

Creating speech zones with self-distributing acoustic swarms

“…The algorithm must be robust to microphone position errors and work across different array shapes and sizes even in reverberant real-world environments. While prior work in deep learning proposed speech separation networks [18][19][20][21] , they did not achieve 2D localization. Recent work also explored distributed microphone arrays 1 .…”

ClearBuds - wireless binaural earbuds for learning-based speech enhancement

“…More recent works tackle the problem of real-time directional hearing using eye trackers and wearable headsets. For example, [49] uses a hybrid network that combines signal processing with neural networks, but shows that their technique performs poorly in binaural scenarios (i.e., two microphones) and requires four or more microphones. In contrast, we focus on the problem of speech enhancement and create the first real-time end-to-end hardware-software neural-network based system using wireless synchronized earbuds.…”