Play as You Like: Timbre-enhanced Multi-modal Music Style Transfer

Crossing You in Style: Cross-modal Style Transfer from Music to Visual Arts

“…Music styles depend on the semantic domain being discussed, such as timbre, performance, or composition styles. Timbre style transfer is usually audio-to-audio style transfer which aims at modifying the timbre such as instrument [9,20,32] or the gender of singers' voice [17,34]. Performance style transfer can be either audio-to-audio or symbolicto-audio, the latter such as piano performance rendering [12,22] refers to the tasks of converting deadpan performance data (e.g., MIDI) into expressive performance with a specific interpretation of timing and dynamics.…”

“…We also introduce an adversarial loss L adv to enhance the training process. Following [20], we employ RaGAN [14] Let P r eal and P дen be the distributions of the image and the music representation, respectively. The adversarial loss is represented as…”

“…Since Gatys et al proposed the neural algorithm for image style transfer [11], deep learning-based style transfer has been extensively studied. Various types of neural network models now can modify the texture of an image [13,19,30], the genre or instrument of a music piece [20,21], and the sentiment of texts [29], with all their content information being preserved. Despite its success, one notable issue in these style transfer methods is that almost all of them operate merely within one single data modality, e.g., from one image to another image, or from one music piece to another.…”