End-to-end Lyrics Alignment for Polyphonic Music Using an Audio-to-character Recognition Model

Abstract

Time-aligned lyrics can enrich the music listening experience by enabling karaoke, text-based song retrieval and intra-song navigation, and other applications. Compared to text-to-speech alignment, lyrics alignment remains highly challenging, despite many attempts to combine numerous sub-modules including vocal separation and detection in an effort to break down the problem. Furthermore, training required fine-grained annotations to be available in some form.Here, we present a novel system based on a modified Wave-U-Net architecture, which predicts character probabilities directly from raw audio using learnt multi-scale representations of the various signal components. There are no sub-modules whose interdependencies need to be optimized. Our training procedure is designed to work with weak, line-level annotations available in the real world. With a mean alignment error of 0.35s on a standard dataset our system outperforms the state-of-the-art by an order of magnitude.

Automatic Lyrics Alignment and Transcription in Polyphonic Music: Does Background Music Help?

“…We compare the performance of a standard ASR trained on extracted singing vocals and polyphonic audio for the tasks of lyrics alignment (Table 3) and transcription ( Table 4). The alignment performance is measured as the mean absolute word boundary error (AE) for each song, averaged over all songs of a dataset, in seconds [12,14], and lyrics transcription performance is measured as the word error rate (WER) which is a standard performance measure for ASR systems. We see an improvement in both alignment and transcription performance with ASR trained on polyphonic data than vocal extracted data, on all the test datasets.…”

Singing Language Identification Using a Deep Phonotactic Approach

“…For the acoustic model F, we rely on a Convolutionnal Recurrent Network (CRNN) trained with the CTC algorithm, as in [16]. CTC-based acoustic models were successfully implemented for singing-related tasks, such as lyrics-to-audio alignment [13,15] and keyword spotting [16]. Following their works, we also employ a singing voice separation preprocessing step during training and inference, which improves performance over using polyphonic data in [13].…”

“…CTC-based acoustic models were successfully implemented for singing-related tasks, such as lyrics-to-audio alignment [13,15] and keyword spotting [16]. Following their works, we also employ a singing voice separation preprocessing step during training and inference, which improves performance over using polyphonic data in [13]. For the recurrent layers, we choose bidirectional Long Short-Term Memory (LSTM) cells to take the full sequence into account when predicting characters at each time frame.…”

“…Recent works have trained new acoustic models with singing data and show great results in lyrics transcription [13], lyrics-to-audio alignment [14,15] and explicit content detection [16], using more recent DNN techniques. In this work, we propose to apply these advances to a phonotactic SLID system: in particular, the usage of the CTC algorithm allows the acoustic model to be trained with DALI, a large multilingual singing dataset [17], while alleviating the need for frame-level aligned lyrics.…”

Weakly Informed Audio Source Separation

“…However, the main obstacle for exploiting side information remains: accurately labeled data is expensive to create and thus rare. For example, aligning a score on the note level or lyrics on the phoneme level would require manual annotations, as creating such fine alignment automatically remains an open problem [15,16]. On the other hand, weak side information such as non-aligned scores or lyrics is often easily available but not straightforward to employ.…”