Multi-Channel Speech Enhancement Using Graph Neural Networks

Tzirakis, Panagiotis; Kumar, Anurag; Donley, Jacob

doi:10.1109/icassp39728.2021.9413955

Cited by 28 publications

(31 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This type of data is the easiest to obtain, since a wide variety of voice types and physical setups can be generated instantly. Many machine learning baselines, e.g., [37,38,69], only train and evaluate on synthetic data generated in this manner. To generate the synthetic dataset, we create multi-speaker recordings in simulated environments with reverb and background noises.…”

Section: Training Methodologymentioning

confidence: 99%

“…Multi-channel source separation and speech enhancement. Multichannel methods have been shown to perform better than their single-channel source separation counterparts [18,[34][35][36][37][38]. Binaural methods have also been used for source separation [39][40][41][42] and localization [43][44][45]; [40] reduces the look-ahead time in the network to make it causal in behavior but has not been demonstrated to run on a mobile device.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

ClearBuds

Chatterjee

Kim

Jayaram

et al. 2022

Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services

View full text Add to dashboard Cite

We present ClearBuds, the first hardware and software system that utilizes a neural network to enhance speech streamed from two wireless earbuds. Real-time speech enhancement for wireless earbuds requires high-quality sound separation and background cancellation, operating in real-time and on a mobile phone. Clear-Buds bridges state-of-the-art deep learning for blind audio source separation and in-ear mobile systems by making two key technical contributions: 1) a new wireless earbud design capable of operating as a synchronized, binaural microphone array, and 2) a lightweight dual-channel speech enhancement neural network that runs on a mobile device. Our neural network has a novel cascaded architecture that combines a time-domain conventional neural network with a spectrogram-based frequency masking neural network to reduce the artifacts in the audio output. Results show that our wireless earbuds achieve a synchronization error less than 64 𝜇s and our network has a runtime of 21.4 ms on an accompanying mobile phone. In-the-wild evaluation with eight users in previously unseen indoor and outdoor multipath scenarios demonstrates that our neural network generalizes to learn both spatial and acoustic cues to perform noise suppression and background speech removal. In a user-study with 37 participants who spent over 15.4 hours rating 1041 audio samples collected in-the-wild, our system achieves improved mean opinion score and background noise suppression. System demo video: https://youtu.be/HYu0ybjcQPA CCS CONCEPTS• Computer systems organization → Embedded systems; • Human-centered computing → Ubiquitous and mobile computing systems and tools; • Computing methodologies → Machine learning.

show abstract

Section: Training Methodologymentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

ClearBuds

Chatterjee

Kim

Jayaram

et al. 2022

Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services

View full text Add to dashboard Cite

show abstract

“…Another work, speech signals are represented as graphs to better capture the global feature representation for speech emotion recognition [21], where deep frame-level features are generated by an LSTM followed by a GNN to classify the graph representation of utterances. In another recent work [22], each audio channel is viewed as a node while constructing a speech graph for speech enhancement task. This allows for the discovery of spatial correlation among several channels.…”

Section: B Graph Neural Network In Audiomentioning

confidence: 99%

Self-supervised Graphs for Audio Representation Learning with Limited Labeled Data

Shirian¹,

Somandepalli²,

Guha³

2022

Preprint

View full text Add to dashboard Cite

Large scale databases with high-quality manual annotations are scarce in audio domain. We thus explore a self-supervised graph approach to learning audio representations from highly limited labeled data. Considering each audio sample as a graph node, we propose a subgraph-based framework with novel self-supervision tasks that can learn effective audio representations. During training, subgraphs are constructed by sampling the entire pool of available training data to exploit the relationship between the labeled and unlabeled audio samples. During inference, we use random edges to alleviate the overhead of graph construction. We evaluate our model on three benchmark audio databases, and two tasks: acoustic event detection and speech emotion recognition. Our semi-supervised model performs better or on par with fully supervised models and outperforms several competitive existing models. Our model is compact (240k parameters), and can produce generalized audio representations that are robust to different types of signal noise.

show abstract

“…A systematic approach to design such networks is to use processing blocks that are number and permutation invariant, such as global pooling and self-attention [13]. Some recent works have investigated DNNs for ad-hoc array processing [11], [14]- [16]. Luo et al [14] proposed a novel transform-average-concatenate module to deal with unknown number and order, and graph neural networks were investigated for distributed arrays by Tzirakis et al [11].…”

Section: Introductionmentioning

confidence: 99%

“…Some recent works have investigated DNNs for ad-hoc array processing [11], [14]- [16]. Luo et al [14] proposed a novel transform-average-concatenate module to deal with unknown number and order, and graph neural networks were investigated for distributed arrays by Tzirakis et al [11]. Wang et al [17] proposed a spatio-temporal network where a recurrent network was used for temporal modeling and selfattention was used for spatial modeling.…”

Section: Introductionmentioning

confidence: 99%

Time-domain Ad-hoc Array Speech Enhancement Using a Triple-path Network

Pandey¹,

Xu²,

Kumar³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Deep neural networks (DNNs) have been successfully used for multichannel speech enhancement in fixed array geometries. However, challenges remain for ad-hoc arrays with unknown microphone placements. We propose a deep neural network based approach for ad-hoc array processing: Triple-Attentive Dual-Recurrent Network (TADRN). TADRN uses self-attention across channels for learning spatial information and a dual-path attentive recurrent network (ARN) for temporal modeling. Temporal modeling is done independently for all channels by dividing a signal into smaller chunks and using an intra-chunk ARN for local modeling and an inter-chunk ARN for global modeling. Consequently, TADRN uses triple-path attention: inter-channel, intra-chunk, and inter-chunk, and dual-path recurrence: intra-chunk and inter-chunk. Experimental results show excellent performance of TADRN. We demonstrate that TADRN improves speech enhancement by leveraging additional randomly placed microphones, even at locations far from the target source. Additionally, large improvements in objective scores are observed when poorly placed microphones in the scene are complemented with more effective microphone positions, such as those closer to a target source.

show abstract

Multi-Channel Speech Enhancement Using Graph Neural Networks

Cited by 28 publications

References 25 publications

ClearBuds

ClearBuds

Self-supervised Graphs for Audio Representation Learning with Limited Labeled Data

Time-domain Ad-hoc Array Speech Enhancement Using a Triple-path Network

Contact Info

Product

Resources

About