Weakly supervised spoken term discovery using cross-lingual side information

Abstract

Recent work on unsupervised term discovery (UTD) aims to identify and cluster repeated word-like units from audio alone. These systems are promising for some very low-resource languages where transcribed audio is unavailable, or where no written form of the language exists. However, in some cases it may still be feasible (e.g., through crowdsourcing) to obtain (possibly noisy) text translations of the audio. If so, this information could be used as a source of side information to improve UTD. Here, we present a simple method for rescoring the output of a UTD system using text translations, and test it on a corpus of Spanish audio with English translations. We show that it greatly improves the average precision of the results over a wide range of system configurations and data preprocessing methods.

Towards speech-to-text translation without speech recognition

“…Poor cross-speaker matches and low audio coverage prevent our system from achieving a high recall, suggesting the of use speech features that are effective in multi-speaker settings (Kamper et al, 2015;Kamper et al, 2016a) and speaker normalization (Zeghidour et al, 2016). Finally, Bansal et al (2017) recently showed that UTD can be improved using the translations themselves as a source of information, which suggests joint learning as an attractive area for future work.…”

“…Many of these errors are due to cross-speaker matches, which are known to be more challenging for UTD (Carlin et al, 2011;Kamper et al, 2015;Bansal et al, 2017). Most matches in our corpus are across calls, yet these are also the least accurate (Table 1).…”

A Small Griko-Italian Speech Translation Corpus

“…For endangered languages (extremely low-resource settings) the lack of training data leads to the problem being framed as a sparse translation problem. This semi-supervised task lies between speech translation and keyword spotting, with cross-lingual supervision being used for word segmentation [30,31,32,33]. Bilingual setups for word segmentation were discussed by [34,35,36,37], but applied to speech transcripts (true phones).…”

Spoken Term Discovery for Language Documentation using Translations

“…We extend s2t to identify new instances of those prototypes in the unlabeled speech, using a modified version of ZRTools, the same UTD toolkit used by UTD-align. 3 (Jansen et al, 2010) Previous work has indicated that using translation text to inform acoustic clustering provides more accurate clusters than just using UTD (Bansal et al, 2017a), so we initially expected that this straightforward extension of s2t would work better than UTD-align. However, early experiments indicated that the text had too much influence on clustering, yielding clusters with highly diverse audio, and thus poor prototypes.…”