CDPAM: Contrastive Learning for Perceptual Audio Similarity

Abstract

Many speech processing methods based on deep learning require an automatic and differentiable audio metric for the loss function. The DPAM approach of Manocha et al.[1] learns a full-reference metric trained directly on human judgments, and thus correlates well with human perception. However, it requires a large number of human annotations and does not generalize well outside the range of perturbations on which it was trained. This paper introduces CDPAM -a metric that builds on and advances DPAM. The primary improvement is to combine contrastive learning and multi-dimensional representations to build robust models from limited data. In addition, we collect human judgments on triplet comparisons to improve generalization to a broader range of audio perturbations. CDPAM correlates well with human responses across nine varied datasets. We also show that adding this metric to existing speech synthesis and enhancement methods yields significant improvement, as measured by objective and subjective tests.

DPLM: A Deep Perceptual Spatial-Audio Localization Metric

“…On the other hand, one may consider adapting existing objective assessment metrics for quality of monaural signals such as PESQ [6], POLQA [7], DPAM [8] and CDPAM [9] for this task. However, since these metrics only focus on perceived quality rather than spatialization, their utility for multi-channel signals remains limited [1,10].…”

Music Enhancement via Image Translation and Vocoding

“…We also compare the subjective listening test scores with widely used audio quality metrics and suggest that, similar to speech enhancement, these metrics correlate poorly with human perception [1,5]. With this work, we hope to motivate both future research in music enhancement as well as music quality perceptual metrics akin to those in the speech literature [6,7]. To promote further research, audio samples generated in our experiments and source code are provided at our project website 1 .…”

UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022

“…Contrastive learning is a self-supervised machine-learning method that can utilize unlabeled data by learning from intrinsic similarity relations between data. Contrastive learning is widely used in speech quality assessment, in which speech representations are learned from large-scale unlabeled speech data [9][10][11]. Given scores s1 and s2 of utterances x1 and x2, the difference in the scores (dx 1 ,x 2 = s1 − s2) can be regarded as the difference in the two utterances in terms of speech quality.…”