Self-Supervised Language Learning From Raw Audio: Lessons From the Zero Resource Speech Challenge

Abstract

Recent progress in self-supervised or unsupervised machine learning has opened the possibility of building a full speech processing system from raw audio without using any textual representations or expert labels such as phonemes, dictionaries or parse trees. The contribution of the Zero Resource Speech Challenge series since 2015 has been to break down this long-term objective into four well-defined tasks-Acoustic Unit Discovery, Spoken Term Discovery, Discrete Resynthesis, and Spoken Language Modeling-and introduce associated metrics and benchmarks enabling model comparison and cumulative progress. We present an overview of the six editions of this challenge series since 2015, discuss the lessons learned, and outline the areas which need more work or give puzzling results.

Word Discovery in Visually Grounded, Self-Supervised Speech Models

“…We then show that these segments can be clustered across a speech corpus to perform syllable discovery, enabling tokenization of the speech signal at the level of syllable-like units. Finally, we also show surprising results where our model trained only on English speech is able to perform zero-shot segmentation of syllables on another language (Estonian) and words in multiple non-English languages, in several cases outperforming the state-of-the-art models on the Zerospeech challenge [13].…”

“…Spoken term discovery -inferring the temporal boundary and identity of words and short phrases from untranscribed speech audio data -has been an important research direction in Zeroresource speech processing [13]. The earliest work that tackles spoken term discovery date back to at least the segmental dynamic programming algorithm proposed by Park and Glass [14].…”

“…To evaluate segmentation performance, we use precision, recall, F1 and R-value [51,23]. For the calculation of above metrics, we use a tolerance window of 50ms for SpokenCOCO and Estonian following [17], and 30ms for the Zerospeech Challenge [13]. To evaluate the quality of our syllable clustering, we first match hypothesized syllable segments with the ground truth segments for each utterance.…”

Word segmentation from transcriptions of child-directed speech using lexical and sub-lexical cues

“…Although this has remained an established paradigm for the study of word segmentation, in recent years the speech research community has made great advances in the area of -  . These studies aim to develop unsupervised methods that learn from raw speech audio only, pioneered in recent years by the Zero Resource Speech Challenge (ZRC) series (Dunbar et al, 2022).…”

“…The latter continue to perform significantly worse than the formerthe top-performing model for the Zero Speech Challenge (ZRC) series segmentation task achieves a token F 1 -score of only 19.2 on the English portion of the TDE-17 test corpus, compared to 64.5 for the text-based topline system provided by the task. In an effort to explain this gap in performance, Dunbar et al (2022) discuss how the higher granularity of analysis, the lack of invariant quantised acoustic representations and the variability of speech rate all contribute. DYMULTI could be run at a higher granularity of analysis, with features extracted directly from the speech stream, to help bridge this gap.…”