The artificial synesthete: Image-melody translations with variational autoencoders

Abstract

This project presents a system of neural networks to translate between images and melodies. Autoencoders compress the information in samples to abstract representation. A translation network learns a set of correspondences between musical and visual concepts from repeated joint exposure. The resulting "artificial synesthete" generates simple melodies inspired by images, and images from music. These are novel interpretation (not transposed data), expressing the machine's perception and understanding. Observing the work, one explores the machine's perception and thus, by contrast, one's own.