Modeling and Training Strategies for Language Recognition Systems

Abstract

Automatic speech recognition is complementary to language recognition. The language recognition systems exploit this complementarity by using frame-level bottleneck features extracted from neural networks trained with a phone recognition task. Recent methods apply frame-level bottleneck features extracted from an end-to-end sequence-to-sequence speech recognition model. In this work, we study an integrated approach of the training of the speech recognition feature extractor and language recognition modules. We show that for both classical phone recognition and end-to-end sequence-to-sequence features, sequential training of the two modules is not the optimal strategy. The feature extractor can be improved by supervision with the language identification loss, either in a fine-tuning step or in a multi-task training framework. Besides, we notice that end-to-end sequence-to-sequence bottleneck features are on par with classical phone recognition bottleneck features without requiring a forced alignment of the signal with target tokens. However, for sequence-to-sequence, the architecture of the model seems to play an important role; the Conformer architectures leads to much better results than the conventional stacked DNNs approach; and can even be trained directly with the LID module in an end-to-end approach.

Three-stage training and orthogonality regularization for spoken language recognition

“…1 in Table 4. We get the same conclusion as [16], that is, finetuning the LID task with the unfrozen encoder outperforms that with the frozen encoder. Note that although No.…”

“…It can be observed that these methods differ in whether the ASR encoder is frozen or not during the second stage of training. It is shown in [16] that the unfrozen encoder is superior in the recognition accuracy. In our preliminary experiments, we tried these two training strategies mentioned above and extracted fixed-length embeddings from some cross channel test data.…”

“…While the classical end-to-end LID methods using acoustic features can implicitly learn phonetic knowledge as well, phonetic features are more easily discovered by an ASR task with frame-level phonemic labels, considering utterance-level linguistic labels might be too coarse [16]. ASR tasks can efficiently extract phonetic features, and noise such as speaker information can be filtered.…”

“…Wang et al [26] analyze different conformer-based architectures and demonstrate the great improvement of a two-stage system, named transfer learning in their experiments. Duroselle et al [16,27] study different modeling and training strategies and show that bottleneck features can be greatly improved by using language identification loss during the training of the feature extractor. Alum et al [28] incorporate a large pre-trained multilingual XLSR-53 wav2vec2.0 model and reveal its excellent modeling abilities, that is, fine-tuning the model with just one utterance per target language already outperforms the baseline model that does not use pre-training but is trained with around 10,000 utterances per language.…”

Differentially Private Speaker Anonymization

“…Bottleneck features. An acoustic model trained for ASR can also be used for other tasks which rely on the phonetic content but do not require a word-level transcription, such as language identification [63] or keyword spotting [77]. In such cases, instead of using the acoustic model output (triphone posterior probabilities), a sequence of phonetic features called bottleneck (BN) features is extracted from an intermediate layer of the acoustic model [81] and used, possibly in combination with other features, as input to these tasks.…”