Unsupervised Phoneme and Word Discovery From Multiple Speakers Using Double Articulation Analyzer and Neural Network With Parametric Bias

Abstract

This paper describes a new unsupervised machine learning method for simultaneous phoneme and word discovery from multiple speakers. Human infants can acquire knowledge of phonemes and words from interactions with his/her mother as well as with others surrounding him/her. From a computational perspective, phoneme and word discovery from multiple speakers is a more challenging problem than that from one speaker because the speech signals from different speakers exhibit different acoustic features. This paper proposes an unsupervised phoneme and word discovery method that simultaneously uses nonparametric Bayesian double articulation analyzer (NPB-DAA) and deep sparse autoencoder with parametric bias in hidden layer (DSAE-PBHL). We assume that an infant can recognize and distinguish speakers based on certain other features, e.g., visual face recognition. DSAE-PBHL is aimed to be able to subtract speaker-dependent acoustic features and extract speaker-independent features. An experiment demonstrated that DSAE-PBHL can subtract distributed representations of acoustic signals, enabling extraction based on the types of phonemes rather than on the speakers. Another experiment demonstrated that a combination of NPB-DAA and DSAE-PB outperformed the available methods in phoneme and word discovery tasks involving speech signals with Japanese vowel sequences from multiple speakers.

Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoring

“…I framed both as instances of a duration-penalized dynamic programming (DPDP) procedure, where a self-supervised neural scoring function is combined with a penalty term that encourages longer segments. I chained the two models to do word segmentation from speech and compared this 6 To get the recall for a particular word type, the number of correctly segmented word tokens (where both boundaries are correct without an intermediate prediction) are divided by the total number of tokens of this type. to existing direct and joint self-supervised approaches on standard benchmarks.…”

“…A segmentation S can be specified as a sequence of (start, end) tuples. The red-solid path in Figure 1 corresponds to the segmentation S = ((1, 3), (4, 4), (5,6), (7,7), (8,9)), giving the result "alo n gt i me". The blue-dashed path is a different segmentation with a different cost.…”

“…Figure 5 shows the token recall for the 15 word types that are most often segmented correctly (top) and the words that are incorrectly segmented most often (bottom). 6 Several of the correctly segmented words are shorter filler words: "mm" and "um" have the highest recall. In contrast, the word types with the lowest recall are longer words.…”

“…In the following decade, several groups moved to joint speech segmentation models where bottom-up information from discovered acoustic units can influence top-down word segmentation and vice versa. The hierarchical Bayesian HMM of Lee et al [5] and the later double articulation models [6], [7] are some examples. Other groups tried to model higher-level E&E Engineering, Stellenbosch University, South Africa.…”

Brain-inspired probabilistic generative model for double articulation analysis of spoken language

“…The language model represents the transition probabilities between words, and the acoustic model represents the relationship between each phoneme and acoustic features. Acoustic features are obtained by dimensional compression of the speech spectrum by conversion to the Mel Frequency Cepstral Coefficient (MFCC) or deep sparse autoencoder with parametric bias in the hidden layer (DSAE-PBHL) [16].…”