Lyrics-to-Audio Alignment by Unsupervised Discovery of Repetitive Patterns in Vowel Acoustics

Abstract

Most of the previous approaches to lyrics-to-audio alignment used a pre-developed automatic speech recognition (ASR) system that innately suffered from several difficulties to adapt the speech model to individual singers. A significant aspect missing in previous works is the self-learnability of repetitive vowel patterns in the singing voice, where the vowel part used is more consistent than the consonant part. Based on this, our system first learns a discriminative subspace of vowel sequences, based on weighted symmetric non-negative matrix factorization (WS-NMF), by taking the self-similarity of a standard acoustic feature as an input. Then, we make use of canonical time warping (CTW), derived from a recent computer vision technique, to find an optimal spatiotemporal transformation between the text and the acoustic sequences. Experiments with Korean and English data sets showed that deploying this method after a pre-developed, unsupervised, singing source separation achieved more promising results than other state-of-the-art unsupervised approaches and an existing ASR-based system.

Contrastive Learning-Based Audio to Lyrics Alignment for Multiple Languages

“…How do similarity models perform on multilingual data? To estimate the multilingual performance of published models trained on English data, we apply the NUS AutoLyricsAlign software 5 on the JamendoLyrics Multi-Lang dataset, and evaluate the alignment 6 . We find that the performance on English matches what was published by the authors [6].…”

“…For many years, the accuracy of systems for the automatic alignment of audio and lyrics content was well below the requirements for practical applications [1][2][3][4][5]. However, since 2019 there has been a resurgence of research and results improved by an order of magnitude.…”

Singing Voice Phoneme Segmentation by Hierarchically Inferring Syllable and Phoneme Onset Positions

“…Chien et al [19] introduced an approach based on vowel likelihood models. Chang and Lee [20] used canonical time warping and repetitive vowel patterns to find the alignment for vowel sequence. Some other works achieved the alignment at music structure-level [21] or line-level [22].…”

A Strategy for Improved Phone-Level Lyrics-to-Audio Alignment for Speech-to-Singing Synthesis

“…Others focus on the alignments of vowels. For instance, the work in [25] combines non-negative matrix factorization and canonical time warping to discover repetitive acoustic patterns of vowels. In [26] the authors proposed to train vowel likelihood models to identify vowels and then to align syllables.…”