Multimodal Music Information Processing and Retrieval: Survey and Future Challenges

Abstract

Towards improving the performance in various music information processing tasks, recent studies exploit different modalities able to capture diverse aspects of music. Such modalities include audio recordings, symbolic music scores, midlevel representations, motion and gestural data, video recordings, editorial or cultural tags, lyrics and album cover arts. This paper critically reviews the various approaches adopted in Music Information Processing and Retrieval, and highlights how multimodal algorithms can help Music Computing applications. First, we categorize the related literature based on the application they address. Subsequently, we analyze existing information fusion approaches, and we conclude with the set of challenges that Music Information Retrieval and Sound and Music Computing research communities should focus in the next years.

What All Do Audio Transformer Models Hear? Probing Acoustic Representations for Language Delivery and Its Structure

“…The temporal feature zero crossing rate measures the rate at which a signal moves from positive to a negative value or vice-versa. It is widely used as a key feature in speech recognition and music information retrieval (Neumayer and Rauber, 2007;Simonetta et al, 2019). Energy features of audio are an important component that characterizes audio signals.…”

Multilayer Music Representation and Processing: Key Advances and Emerging Trends

“…Reference [2] represented a critical survey on multimodal collaborative processing and retrieval of music information. The goal was to highlight how multimodal algorithms, working simultaneously on audio and video recordings, symbolic music scores, mid-level representations, motion and gestural data, etc., can help Music Computing applications.…”

“…In conclusion, session Representations provided multifaceted views on the complex issue of digital music representation, describing the features of current formats [4] and proposing new ones [1], highlighting how a suitable representation can turn into an effective way to improve music information computing and retrieval [2], and proposing new forms of semantic representation of music items [3].…”

Judgement of valence of musical sounds by hand and by heart, a machine learning paradigm for reading the heart

“…Thus, music has been studied from several perspectives. For instance, in the direction of multimodal music information, several tasks have been developed, among them it is possible to find music segmentation, emotion or mood recognition, synchronization of different representations, and classification of music [ 22 ]. In like manner, some researchers have developed systems based on deep learning to create music [ 23 ].…”