Domain-Informed Probing of wav2vec 2.0 Embeddings for Phonetic Features

Abstract

In recent years large transformer model architectures have become available which provide a novel means of generating high-quality vector representations of speech audio. These transformers make use of an attention mechanism to generate representations enhanced with contextual and positional information from the input sequence. Previous works have explored the capabilities of these models with regard to performance in tasks such as speech recognition and speaker verification, but there has not been a significant inquiry as to the manner in which the contextual information provided by the transformer architecture impacts the representation of phonetic information within these models. In this paper, we report the results of a number of probing experiments on the representations generated by the wav2vec 2.0 model's transformer component, with regard to the encoding of phonetic categorization information within the generated embeddings. We find that the contextual information generated by the transformer's operation results in enhanced capture of phonetic detail by the model, and allows for distinctions to emerge in acoustic data that are otherwise difficult to separate.

Phonemic competition in end-to-end ASR models

“…The results in the previous sections show that MLP-based classifiers perform adequately on the latent representations for all layers (Table 1, in line with [2,3]), while the internal phonephone structure shows a varying pattern across layers (Figs. 1, 2).…”

“…Importantly, this does not mean that those representations contain weaker phonetic information. After all, [2,3,4] have shown that classifiers that use these representations for a range of downstream tasks outperform the results of similar classifiers that operate on conventional spectral representations. The figure shows that the acoustic organisation in the higher transformer layers differs from the lowest transformer layer, in line with the findings in the next sections.…”

“…In our second method we trained MLP-based phone classifiers similar to the approach reported in other probing papers (e.g. [3]). First, we used both a pre-trained Wav2vec2 model without any fine-tuning, and a fine-tuned model trained on the core part of the read aloud books component (component o) in the Spoken Dutch Corpus [14].…”

“…The recent emergence of so-called end-to-end systems (E2E), such as Wav2Vec 2.0 [1] (henceforth Wav2vec2), has revolutionized automatic speech recognition (ASR) in many ways. At the same time many researchers whose interest is not primarily to obtain the lowest possible transcription error rate are asking whether the representations on some or all layers of E2E models contain information that can be harnessed for other downstream tasks [2,3,4]. These approaches are collectively known as probing.…”

“…Many probing approaches take the representations on some layer of a deep neural network (DNN) as the data with which some classifier is trained for some specific task, such as phonetic feature extraction of phone classification. Previous research such as [2,3,4,9] mainly focused on identifying the layer whose representations yielded the best classification performance. These studies show that phone classification is possible with convincing performance.…”