Evaluating Creative Language Generation: The Case of Rap Lyric Ghostwriting

Abstract

Language generation tasks that seek to mimic human ability to use language creatively are difficult to evaluate, since one must consider creativity, style, and other non-trivial aspects of the generated text. The goal of this paper is to develop evaluation methods for one such task, ghostwriting of rap lyrics, and to provide an explicit, quantifiable foundation for the goals and future directions of this task. Ghostwriting must produce text that is similar in style to the emulated artist, yet distinct in content. We develop a novel evaluation methodology that addresses several complementary aspects of this task, and illustrate how such evaluation can be used to meaningfully analyze system performance. We provide a corpus of lyrics for 13 rap artists, annotated for stylistic similarity, which allows us to assess the feasibility of manual evaluation for generated verse.

Dyr Bul Shchyl. Proxying Sound Symbolism With Word Embeddings

“…Applying the obtained sound symbolism information to generative tasks, one can expect to generate more expressive poetry in line with the results of (Auracher et al, 2010). This new approach combined with such generative methods as (Potash et al, 2016), (Tikhonov and Yamshchikov, 2018), (Vechtomova et al, 2018) or (Wołk et al, 2019). The possibility of testing specific associations between sounds and semantics computationally without any behavioral laboratory experiments or surveys might also significantly facilitate further studies of semantic symbolism.…”

Creative GANs for generating poems, lyrics, and metaphors

“…Evaluating creative generation tasks is both critical and complex [27]. Along the lines of previous research on evaluating text generation tasks [27], we report the perplexity scores of our test set on the evaluated models in the Supplementary Section, Table 1 Our model shows improvements over baseline and GumbelGAN.…”

“…Evaluating creative generation tasks is both critical and complex [27]. Along the lines of previous research on evaluating text generation tasks [27], we report the perplexity scores of our test set on the evaluated models in the Supplementary Section, Table 1 Our model shows improvements over baseline and GumbelGAN. Common computational methods like BLEU [28] and perplexity are at best a heuristic and not strong indicators of good performance in text generation models [29].…”

Youling: an AI-assisted Lyrics Creation System

“…However, despite the popularity of lyrics generation, there still lacks a comprehensive lyrics creation assistant system for music creators. Previous researches (Castro and Attarian, 2018;Saeed et al, 2019;Lu et al, 2019;Manjavacas et al, 2019;Watanabe et al, 2018;Potash et al, 2018;Fan et al, 2019;Li et al, 2020) and systems (Potash et al, 2015;Lee et al, 2019;Shen et al, 2019), are mostly model-oriented, utilizing * Equal contribution † Corresponding Author neural networks including GAN, RNN-based or Transformer-based (Vaswani et al, 2017) sequence to sequence (Seq2Seq) models for sentence-wise lyrics generation. They complete the lyrics generation process in a single pass with specific keywords or content controlling attributes as input, involving little human intervention.…”