Speech acoustic modeling from raw multichannel waveforms

Abstract

Standard deep neural network-based acoustic models for automatic speech recognition (ASR) rely on hand-engineered input features, typically log-mel filterbank magnitudes. In this paper, we describe a convolutional neural network -deep neural network (CNN-DNN) acoustic model which takes raw multichannel waveforms as input, i.e. without any preceding feature extraction, and learns a similar feature representation through supervised training.By operating directly in the time domain, the network is able to take advantage of the signal's fine time structure that is discarded when computing filterbank magnitude features. This structure is especially useful when analyzing multichannel inputs, where timing differences between input channels can be used to localize a signal in space. The first convolutional layer of the proposed model naturally learns a filterbank that is selective in both frequency and direction of arrival, i.e. a bank of bandpass beamformers with an auditory-like frequency scale. When trained on data corrupted with noise coming from different spatial locations, the network learns to filter them out by steering nulls in the directions corresponding to the noise sources. Experiments on a simulated multichannel dataset show that the proposed acoustic model outperforms a DNN that uses log-mel filterbank magnitude features under noisy and reverberant conditions.

Deep Convolutional Neural Networks for Predominant Instrument Recognition in Polyphonic Music

“…In addition, our research also differs from [28] because we use single-label data for training and estimate multi-label data, while they used multilabel data from the training phase. Moreover, they focused on an end-to-end approach, which is promising in that using raw audio signals makes the system rely less on domain knowledge and preprocessing, but usually it shows a slightly lower performance than using spectral input such as melspectrogram in recent papers [29], [30].…”

Enhancing End-to-End Multi-Channel Speech Separation Via Spatial Feature Learning

“…To perform the spatial feature learning jointly with the rest of the network, we propose to learn spatial features directly from multi-channel waveforms with an integrated architecture. The main idea is to learn time-domain filters spanning all signal channels to perform adaptive spatial filtering [13][14][15]. These filters parameters are jointly optimized with the encoder using Eq.…”

Deep unfolding for multichannel source separation

“…When A (k) = 0 and c (k) = log π j,z for all k, the new update (16) simplifies to the original variational update (9). Although the synchronous mean-field updates break the variational bound, we expect discriminative training to compensate such approximations.…”

“…Swietojanski et al [15] proposed a convolutional neural network (CNN) architecture for ASR using multichannel audio, where different microphone channels were pooled together. Hoshen et al [16] used a CNN-DNN for acoustic modeling on raw time-domain multichannel audio. Nugraha et al [17] achieved improved source separation for two-channel music recordings using alternating ReLU layers and channel estimation.…”

“…The GMM parameters are discriminatively trained jointly with the DNN parameters for ASR. Hoshen et al's approach [16] attempts to mimic the usual feature extraction pipeline in ASR. However, all these methods suffer the same drawback: the optimal network architectures can only be discovered by heuristic experimentation, and it is difficult to directly incorporate insight from domain knowledge.…”