Zero-Shot Personalized Speech Enhancement Through Speaker-Informed Model Selection

Personalized Neural Speech Codec

“…Third, when we compare the two model sizes, more significant performance improvement is observed when smaller models are in comparison (PLPCNet-S vs. LPCNet-BL-S) than the larger models ((PLPCNet-L vs. LPCNet-BL-L). This trend aligns well with the personalized speech enhancement literature: model personalization benefits compressed model architectures more than the larger ones [15,16,17]. Finally, it is also worth noting that each test sequence is handled by a selected personalized decoder, where the choice is based on the estimated speaker class.…”

“…The Siamese network F(•) employs two 32-unit GRU layers as proposed in [16], while we train it without any noise injection. Fig.…”

“…The subplots show that learned embedding space provides perceptually meaningful discrimination of speakers, e.g., when C = 2 the clusters are formed by the gender of the speakers. In theory, more clusters could lead to better specialization, while the performance gain achieved by personalization saturates at some point, too, as shown in [16]. Considering the number of speakers and amount of utterances in each speaker group, we choose to partition the training set into four groups, i.e., C = 4.…”

“…Likewise, a reasonable grouping strategy of speakers is key to successful personalization. Following [16], we first learn the speaker encoder F(•) that learns discriminative speaker embeddings z. They define the embedding space, where clustering is performed.…”

“…Personalization has shown promising results in model compression tasks for speech enhancement [15,16,17,18]. A personalized model adapts to the target speaker group's speech trait, narrowing the training task down to a smaller subtask, i.e., defined by the smaller speaker group than the entire speakers in the corpus.…”