Segmental Contrastive Predictive Coding for Unsupervised Word Segmentation

Abstract

Automatic detection of phoneme or word-like units is one of the core objectives in zero-resource speech processing. Recent attempts employ self-supervised training methods, such as contrastive predictive coding (CPC), where the next frame is predicted given past context. However, CPC only looks at the audio signal's frame-level structure. We overcome this limitation with a segmental contrastive predictive coding (SCPC) framework that can model the signal structure at a higher level e.g. at the phoneme level. In this framework, a convolutional neural network learns frame-level representation from the raw waveform via noise-contrastive estimation (NCE). A differentiable boundary detector finds variable-length segments, which are then used to optimize a segment encoder via NCE to learn segment representations. The differentiable boundary detector allows us to train frame-level and segment-level encoders jointly. Typically, phoneme and word segmentation are treated as separate tasks. We unify them and experimentally show that our single model outperforms existing phoneme and word segmentation methods on TIMIT and Buckeye datasets. We analyze the impact of boundary threshold and when is the right time to include the segmental loss in the learning process.

Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoring

“…Table I shows that the DPDP approach outperforms [25], [26], but performs slightly worse on most metrics compared to the recent state-of-the-art unsupervised phone segmentation approaches [13], [14], [38]. Of the three DPDP systems, the CPC+K-means model (here used as a DPDP scoring network for the first time) achieves the best precision and R-value, V. DPDP FOR UNSUPERVISED WORD SEGMENTATION FROM SYMBOLIC INPUT The goal in unsupervised word segmentation from symbolic input is to break up an input sequence (normally of phonemes or phones) into subsequences representing words.…”

“…The idea is that a model would need to learn meaningful phonetic contrasts while being invariant to nuisance factors such as speaker. Bhati et al [13] extend this by using a second segment-level CPC layer. The segmental CPC (SCPC) consists of a frame-level CPC module, a differentiable boundary detector operating on the learned features, and a segment-level CPC module operating on aggregated features from the lower layer.…”

“…E.g., the Bayesian embedded segmental GMM (BES-GMM) [9] performs probabilistic clustering and segmentation on fixed-dimensional representations of whole word-like segments, as does the related embedded segmental K-means (ES-KMeans) model [10]. Much more recently, self-supervised neural networks using contrastive predictive coding (CPC) [12] have been modified to include separate acoustic unit and word segmentation modules that are trained jointly [13], [14]. These joint models give state-of-theart word segmentation results (more details in §II).…”

“…Given that direct word modelling [9], [10] and joint phone and word modelling [13], [14] have proven to be such fruitful directions for speech segmentation, maybe the original premise of [1] was flawed? This paper argues that their idea-where bottom-up acoustic unit discovery and word segmentation are performed separately-should be revisited.…”

“…For symbolic word segmentation, DPDP is applied with an autoencoding recurrent neural network (AE-RNN), an approach inspired by [19]. Chaining these two DPDP models gives similar performance to the joint self-supervised segmental [13] and aligned [14] CPC models on English data. The DPDP system also achieves the best reported word segmentation scores on the French and Mandarin ZeroSpeech 2017 and 2020 benchmarks [16], [20].…”