Conditional LSTM-GAN for Melody Generation from Lyrics

Yu, Yi; Srivastava, Abhishek; Canales, Simon

doi:10.1145/3424116

Cited by 72 publications

(19 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With an appropriate generator architecture, GANs can be used to generate sequential data such as audio and music as well. They are now able to generate melodies from lyrics (Yu et al, 2021) or compose music from a single latent vector (Engel et al, 2019), or other synthetic time-series data like electrocardiograms (Zhu et al, 2019b), stock market trends, and electricity consumption (Yoon et al, 2019).…”

Section: Generative Adversarial Networkmentioning

confidence: 99%

GANmapper: geographical data translation

Wu,

Biljecki

2021

Preprint

View full text Add to dashboard Cite

We present a new method to create spatial data using a generative adversarial network (GAN). Our contribution uses coarse and widely available geospatial data to create maps of less available features at the finer scale in the built environment, bypassing their traditional acquisition techniques (e.g. satellite imagery or land surveying). In the work, we employ land use data and road networks as input to generate building footprints, and conduct experiments in 9 cities around the world. The method, which we implement in a tool we release openly, enables generating approximate maps of the urban form, and it is generalisable to augment other types of geoinformation, enhancing the completeness and quality of spatial data infrastructure. It may be especially useful in locations missing detailed and high-resolution data and those that are mapped with uncertain or heterogeneous quality, such as much of OpenStreetMap. The quality of the results is influenced by the urban form and scale. In most cases, experiments suggest promising performance as the method tends to truthfully indicate the locations, amount, and shape of buildings. The work has the potential to support several applications, such as energy, climate, and urban morphology studies in areas previously lacking required data.

show abstract

Section: Generative Adversarial Networkmentioning

confidence: 99%

GANmapper: geographical data translation

Wu,

Biljecki

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Automatic songwriting has always been the dream of human beings. In recent years, with the development of artificial intelligence, researchers have achieved great success in various aspects of automatic songwriting such as lyric generation [12,20], melody generation [19,23], lyric-to-melody generation [2,5,16,21], and melodyto-lyric generation [8,11,20]. Among all directions, lyric-to-melody generation is one of the most fundamental tasks and is the focus of this paper.…”

Section: Introductionmentioning

confidence: 99%

Re-creation of Creations: A New Paradigm for Lyric-to-Melody Generation

Lv¹,

Xu²,

Qin³

et al. 2022

Preprint

View full text Add to dashboard Cite

Lyric-to-melody generation is an important task in songwriting, and is also quite challenging due to its distinctive characteristics: the generated melodies should not only follow good musical patterns, but also align with features in lyrics such as rhythms and structures. These characteristics cannot be well handled by neural generation models that learn lyric-to-melody mapping in an endto-end way, due to several issues: (1) lack of aligned lyric-melody training data to sufficiently learn lyric-melody feature alignment; (2) lack of controllability in generation to explicitly guarantee the lyricmelody feature alignment. In this paper, we propose Re-creation of Creations (ROC), a new paradigm for lyric-to-melody generation that addresses the above issues through a generation-retrieval pipeline. Specifically, our paradigm has two stages: (1) creation stage, where a huge amount of music pieces are generated by a neural-based melody language model and indexed in a database through several key features (e.g., chords, tonality, rhythm, and structural information including chorus or verse); (2) re-creation stage, where melodies are recreated by retrieving music pieces from the database according to the key features from lyrics and concatenating best music pieces based on composition guidelines and melody language model scores. Our new paradigm has several advantages: (1) It only needs unpaired melody data to train melody language model, instead of paired lyric-melody data in previous models. (2) It achieves good lyric-melody feature alignment in lyric-to-melody generation. Experiments on English and Chinese datasets demonstrate that ROC outperforms previous neural based lyric-to-melody generation models on both objective and subjective metrics. Demos are in https://ai-muzic.github.io/roc, and the code is in https://github.com/microsoft/muzic/.

show abstract

“…Automatic song writing is an interesting and challenging task in both research and industry. Two most important tasks in automatic song writing are lyric-to-melody generation (L2M) (Bao et al 2019;Yu and Canales 2019;Lee, Fang, and Ma 2019) and melody-to-lyric generation (M2L) (Watanabe et al 2018;Lu et al 2019;Lee, Fang, and Ma 2019). L2M and M2L can be regarded as sequence to sequence learning tasks and can be modeled by the techniques in natural language processing since both melody and lyric can be represented as discrete token sequence.…”

Section: Introductionmentioning

confidence: 99%

“…2 Background Automatic Song Writing Automatic song writing usually covers several tasks including lyric generation (Malmi et al 2015), melody generation (Zhu et al 2018), lyricto-melody generation (L2M) (Choi, Fazekas, and Sandler 2016;Yu and Canales 2019) and melody-to-lyric generation (M2L) (Bao et al 2019;Li et al 2020). In this work, we focus on L2M and M2L.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint

Sheng¹,

Song²,

Tan³

et al. 2020

Preprint

View full text Add to dashboard Cite

Automatic song writing aims to compose a song (lyric and/or melody) by machine, which is an interesting topic in both academia and industry. In automatic song writing, lyric-tomelody generation and melody-to-lyric generation are two important tasks, both of which usually suffer from the following challenges: 1) the paired lyric and melody data are limited, which affects the generation quality of the two tasks, considering a lot of paired training data are needed due to the weak correlation between lyric and melody; 2) Strict alignments are required between lyric and melody, which relies on specific alignment modeling. In this paper, we propose SongMASS to address the above challenges, which leverages masked sequence to sequence (MASS) pre-training and attention based alignment modeling for lyric-to-melody and melody-to-lyric generation. Specifically, 1) we extend the original sentence-level MASS pre-training to song level to better capture long contextual information in music, and use a separate encoder and decoder for each modality (lyric or melody); 2) we leverage sentence-level attention mask and token-level attention constraint during training to enhance the alignment between lyric and melody. During inference, we use a dynamic programming strategy to obtain the alignment between each word/syllable in lyric and note in melody. We pre-train SongMASS on unpaired lyric and melody datasets, and both objective and subjective evaluations demonstrate that SongMASS generates lyric and melody with significantly better quality than the baseline method without pretraining or alignment constraint.

show abstract

Conditional LSTM-GAN for Melody Generation from Lyrics

Cited by 72 publications

References 13 publications

GANmapper: geographical data translation

GANmapper: geographical data translation

Re-creation of Creations: A New Paradigm for Lyric-to-Melody Generation

SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint

Contact Info

Product

Resources

About