“A Good Algorithm Does Not Steal – It Imitates”: The Originality Report as a Means of Measuring When a Music Generation Algorithm Copies Too Much

Abstract

Research on automatic music generation lacks consideration of the originality of musical outputs, creating risks of plagiarism and/or copyright infringement. We present the originality report -a set of analyses for measuring the extent to which an algorithm copies from the input music on which it is trained. First, a baseline is constructed, determining the extent to which human composers borrow from themselves and each other in some existing music corpus. Second, we apply a similar analysis to musical outputs of runs of MAIA Markov and Music Transformer generation algorithms, and compare the results to the baseline. Third, we investigate how originality varies as a function of Transformer's training epoch. Results from the second analysis indicate that the originality of Transfomer's output is below the 95%-confidence interval of the baseline. Musicological interpretation of the analyses shows that the Transformer model obtained via the conventional stopping criteria produces single-note repetition patterns, resulting in outputs of low quality and originality, while in later training epochs, the model tends to overfit, producing copies of excerpts of input pieces. We recommend the originality report as a new means of evaluating algorithm training processes and outputs in future, and question the reported success of language-based deep learning models for music generation. Supporting materials (code, dataset) will be made available via https://osf.io/96emr/.

Deep learning’s shallow gains: a comparative evaluation of algorithms for automatic music generation

“…Some participants indicated this excerpt contains too many repetitions, while others appreciated the crossing of the voices, and the overall sonic effect. Objectively, it highlights the ability of MuTr to Deep learning systems suffer from copying large chunks of the training set in their outputs (Yin et al, 2021). This is the case here.…”

“…Thus, we applied some automated filters to address (a), (b), and (c), as otherwise we would have been wasting participants' time with these excerpts. The filters exclude generated excerpts with a large number of repeated notes or long rests, and we then apply the originality report method (Yin et al, 2021) to measure and exclude excerpts that copy too much input in their output. The effect of these filters is non-negligible, with empirical probability of removal in the range 0.05-0.1, depending on filter and model.…”

“…6 copying the motif of "Carol of the Bells" composed by Mykola Leontovych (1914). This underlines the need for a systematic and widely adopted method to identify copying and potential copyright infringement in the output of AMG systems, for example, the originality report of Yin et al (2021).…”

“…An ethical issue here is that the data-driven property of generative models can lead to those systems copying chunks from original music data, which may result in plagiarism or copyright infringement. According to our previous work (Yin et al, 2021), existing deep learning AMG research neglects to check copying as part of its evaluation process. In that work, we discuss the nature of the language model-based Music Transformer algorithm, and show the extent of replication through various training checkpoints.…”

“…In that work, we discuss the nature of the language model-based Music Transformer algorithm, and show the extent of replication through various training checkpoints. For the listening study, we apply the same method (Yin et al, 2021) to measure and exclude excerpts that copy too much from the training set. The originality baseline, showing percentage of copying, is constructed with training and validation sets, and any generated excerpt with originality less than the lower 95%-confidence interval of the baseline is removed.…”