Hidden Markov Model Variational Autoencoder for Acoustic Unit Discovery

Abstract

In this paper we present our system for the detection and classification of acoustic scenes and events (DCASE) 2020 Challenge Task 4: Sound event detection and separation in domestic environments. We introduce two new models: the forward-backward convolutional recurrent neural network (FBCRNN) and the tagconditioned convolutional neural network (CNN). The FBCRNN employs two recurrent neural network (RNN) classifiers sharing the same CNN for preprocessing. With one RNN processing a recording in forward direction and the other in backward direction, the two networks are trained to jointly predict audio tags, i.e., weak labels, at each time step within a recording, given that at each time step they have jointly processed the whole recording. The proposed training encourages the classifiers to tag events as soon as possible. Therefore, after training, the networks can be applied to shorter audio segments of, e.g., 200 ms, allowing sound event detection (SED). Further, we propose a tag-conditioned CNN to complement SED. It is trained to predict strong labels while using (predicted) tags, i.e., weak labels, as additional input. For training pseudo strong labels from a FBCRNN ensemble are used. The presented system scored the fourth and third place in the systems and teams rankings, respectively. Subsequent improvements allow our system to even outperform the challenge baseline and winner systems in average by, respectively, 18.0 % and 2.2 % event-based F1-score on the validation set.

Unsupervised Speech Representation Learning Using WaveNet Autoencoders

“…Extending it with an HMM temporal structure for sub-phonetic units leads to the DP-HMM and the HDP-HMM [69], [70], [71]. HMM-VAE proposes the use of a deep neural network instead of a GMM [72], [73]. These approaches enforce top-down constraints via HMM temporal smoothing and temporal modeling.…”

Bayesian Subspace Hidden Markov Model for Acoustic Unit Discovery

“…We evaluated the different AUD algorithms in terms of phonetic segmentation and equivalent Phone Error Rate (eq. PER) ( [17,5]). For the phonetic segmentation we used the standard Recall, Precision and F-score measured against the timing provided in the TIMIT database with the 61 original phones.…”

“…Prior Language Recall Precision F-score eq. PER HMM [5] MFCC + ∆ + ∆∆ None ---65.4 VAE-HMM [5] MFCC + ∆ + ∆∆ None ---58.9 VAE-BHMM [6] iterations. Results, shown in Fig.…”

“…In this experiment, all the HMM/SHMM have 4 Gaussian components per state. Our baseline is the HMM based AUD system described in [3] and the VAE-(B)HMM based AUD system proposed in [5,6]. We compare these baselines with 3 SHMM based AUD models for which the posterior of the phonetic subspace q(W, b) was estimated using: (i) German language (ii) German and Polish languages (iii) German, Polish and Spanish languages.…”

Bayesian Models for Unit Discovery on a Very Low Resource Language

“…It was also shown that this model can be further improved by incorporating a Bayesian "phonotactic" language model learned jointly with the acoustic units [4]. Finally, following the work in [5] it has been combined successfully with variational auto-encoders leading to a model combining the potential of both deep neural networks and Bayesian models [6]. The contribution of this work is threefold:…”