Automatic Lyrics Alignment and Transcription in Polyphonic Music: Does Background Music Help?

Abstract

Background music affects lyrics intelligibility of singing vocals in a music piece. Automatic lyrics alignment and transcription in polyphonic music are challenging tasks because the singing vocals are corrupted by the background music. In this work, we propose to learn music genre-specific characteristics to train polyphonic acoustic models. We first compare several automatic speech recognition pipelines for the application of lyrics transcription. We then present the lyrics alignment and transcription performance of musicinformed acoustic models for the best-performing pipeline, and systematically study the impact of music genre and language model on the performance. With such genre-based approach, we explicitly model the music without removing it during acoustic modeling. The proposed approach outperforms all competing systems in the lyrics alignment and transcription tasks on several well-known polyphonic test datasets.

Low Resource Audio-To-Lyrics Alignment from Polyphonic Music Recordings

“…Then a forward-pass decoding algorithm is applied on these posteriograms, obtaining phoneme alignments. Then using a language model (LM), phoneme posteriograms can be converted to word posteriograms to retrieve word-level alignments [1, 3,4]. One recent successful system [2] showed a considerable performance boost compared to previous research using an end-to-end approach trained on a large corpus, where alphabetic characters are used as sub-word units of speech.…”

“…In addition, the authors have used a public dataset [9] that is much smaller than the training set used in [2]. Gupta et al [3] reported state-of-the-art results using an acoustic model trained on polyphonic music using genre-specific phonemes. According to the authors, their system applies forced alignment with a large beam size as their system attempts to process the entire music recording at once.…”

“…VA [5] also uses an end-to-end model, but trained on the DALI (v1.0) dataset, which has over 200 hours of polyphonic music recordings and extracts the vocals using Spleeter. GC1 [3] uses the same training data, for constructing an acoustic model in the hybrid-ASR setting [25] and performs alignment on the original polyphonic mix as well. In addition to these models, we refer to our models which align words to the vocal tracks separated by Demucs and Spleeter as DE1 and DE2 respectively.…”

“…The task of aligning song lyrics with their corresponding music recordings is among the most challenging tasks in music information retrieval (MIR) research due to three major factors: the multimodality of the information to be processed -namely music and speech, the presence of the musical accompaniment in the acoustic scene and the length of the music recording to be aligned. For processing linguistically relevant information, previous studies have taken the approach of adapting automatic speech recognition (ASR) paradigms to singing voice signals [1,2,3,4]. Regarding the musical accompaniment, researchers have aligned lyrics on either source separated vocal tracks [4] or utilized acoustic models trained on polyphonic recordings [2,3,5].…”

“…For processing linguistically relevant information, previous studies have taken the approach of adapting automatic speech recognition (ASR) paradigms to singing voice signals [1,2,3,4]. Regarding the musical accompaniment, researchers have aligned lyrics on either source separated vocal tracks [4] or utilized acoustic models trained on polyphonic recordings [2,3,5].…”