LyricSynchronizer: Automatic Synchronization System Between Musical Audio Signals and Lyrics

Abstract

Abstract-This paper describes a system that can automatically synchronize polyphonic musical audio signals with their corresponding lyrics. Although methods for synchronizing monophonic speech signals and corresponding text transcriptions by using Viterbi alignment techniques have been proposed, these methods cannot be applied to vocals in CD recordings because vocals are often overlapped by accompaniment sounds. In addition to a conventional method for reducing the influence of the accompaniment sounds, we therefore developed four methods to overcome this problem: a method for detecting vocal sections, a method for constructing robust phoneme networks, a method for detecting fricative sounds, and a method for adapting a speech-recognizer phone model to segregated vocal signals. We then report experimental results for each of these methods and also describe our music playback interface that utilizes our system for synchronizing music and lyrics.

Singing information processing

“…When compared to lyric transcription, lyric synchronization presents a simpler prob lem so it can be achieved at a practical level of performance with relative ease and, except for some cases [40], it can target singing with musical accompaniment. Research into lyric synchronization targeting singing with accompaniment can be divided into two cate gories: that using no forced alignment [41][42][43] and that using forced alignment [44][45][46][47][48] . When textual chord information is additionally given, it can be used to improve the accuracy of lyric synchroniza tion [49].…”

Leveraging repetition for improved automatic lyric transcription in popular music

“…Perhaps due to the challenging nature of performing full transcription of the sung voice, researchers have mostly in the past concentrated on the task of aligning/synchronising lyrics to audio, where the task is to assign timestamps to a set of lyrics given the corresponding audio (see, for example, [12,[17][18][19][20]). However, there are clearly situations in which ALR is required.…”

“…However, the majority of these studies assume that the lyrics to a song are known in advance. The reason for this is clear: despite huge advances in Automatic Speech Recognition (ASR), Automatic Lyric Recognition (ALR) is a challenging problem, in part due to the background musical accompaniment and low similarity between the spoken and sung voices [7,[9][10][11][12][13][14].…”

Towards Cross-Version Harmonic Analysis of Music

“…It is equally applicable to general multimedia data, where several versions or representations are given for an object to be analyzed. For example, a robust alignment between given music recordings and lyrics allows for creating karaoke applications [2], [3] or for combining genre classification results across the audio and the text domain [4]. Similarly, combining web-based text information, symbolic music representations and audio data was shown to lead to significant performance gains for general music classification tasks [5].…”

“…As a last stage, key can produce chord labels and a roman numeral analysis, an analysis describing the relation between a chord and the key of the segment the chord is part of. 2 For the evaluation described below, we made use of the information about chord root, mode (major, minor, unspecified) and fifth (perfect, diminished, unspecified) as well as the onset and offset times. This leads to three possible chord classes, namely major, minor, and diminished.…”