Weak top-down constraints for unsupervised acoustic model training

Abstract

Typical supervised acoustic model training relies on strong top-down constraints provided by dynamic programming alignment of the input observations to phonetic sequences derived from orthographic word transcripts and pronunciation dictionaries. This paper investigates a much weaker form of top-down supervision for use in place of transcripts and dictionaries in the zero resource setting. Our proposed constraints, which can be produced using recent spoken term discovery systems, come in the form of pairs of isolated word examples that share the same unknown type. For each pair, we perform a dynamic programming alignment of the acoustic observations of the two constituent examples, generating an inventory of cross-speaker frame pairs that each provide evidence that the same subword unit model should account for them. We find these weak top-down constraints are capable of improving model speaker independence by up to 57% relative over bottom-up training alone.

A summary of the 2012 JHU CLSP workshop on zero resource speech technologies and models of early language acquisition

“…Likewise, stripped of any guidance from word transcripts and a pronunciation dictionary, the normal expectation-maximization training procedures for Gaussian mixturebased acoustic models are no longer capable of identifying speaker independent phonetic categories in a purely bottom-up fashion [24]. With these considerations in mind, an explicit goal of the workshop was to evaluate a variety of acoustic front-ends and unsupervised acoustic modeling strategies, both in isolation and in combination, for suitability in downstream zero resource technologies.…”

“…This is a much weaker form of supervision, but it comes at little cost. The approach considered in the workshop, described in detail in [24], consists of four steps: (1) training a 1024-component Gaussian mixture model (GMM) on a large sample of indomain audio, which serves as a sort of universal background model (UBM) for all speech sounds; (2) running a spoken term discovery system across the speech collection to produce a collection of word or phrase segment pairs and compute UBM posteriorgrams for each segment; (3) performing a DTW alignment of the acoustic frames of each word segment pair and use the frame-level correspondences to construct a similarity matrix over UBM components; and (4) partitioning the UBM Gaussian components with spectral clustering [26] and using each subset to define a subword unit GMM.…”

Partitioning of Posteriorgrams Using Siamese Models for Unsupervised Acoustic Modelling

“…In the first case, acoustic models are inferred directly from the acoustic features [14]- [21]. The second approach is to first segment the speech into syllable-or word-like units, and afterwards break these units into smaller subword units [7], [13], [19], [22]- [29].…”

“…The states from each HMM are then clustered based on the similarity of their distributions, to form subword unit candidates. A related approach is taken in [22], where instead of HMM states, components from a GMM trained on speech frames are clustered based on co-occurence in pairs of fragments obtained from UTD. A neural network referred to as the ABnet and based on siamese networks [30] is introduced in [25].…”

“…This way we are able to take advantage of both the whole unlabelled data set, and the smaller set of fragments discovered by UTD. While the approach of partitioning posteriorgrams is reminiscent of [22], the major difference is that in place of direct clustering of classes, we are instead trying to maximise the similarity/dissimilarity between pairs of speech fragments, which only indirectly results in a partition of the classes. Our linear model also has the advantage of being more interpretable than deep networks like that of [26].…”