Document dependent fusion in multimodal music retrieval

Abstract

In this paper, we propose a novel multimodal fusion framework, document dependent fusion (DDF), which derives the optimal combination strategy for each individual document in the fusion process. For each document, we derive a document weight vector by estimating the descriptive abilities of its different modalities. The document weight vector also enables our framework to be easily integrated with existing multimodal fusion schemes, and achieve a better combination strategy for each document given a query. Experiments are conducted on a 17174-song music database to compare the retrieval accuracy of traditional query independent fusion and query dependent fusion approaches, and that obtained after integrating DDF with them. Experimental results indicate that DDF can significantly improve the retrieval performance of current fusion approaches.

Virtual Instrument Performances (VIP): A Comprehensive Review

“…The first category of methods utilized Long Short‐Term Memory networks (LSTMs), mainly due to their capabilities in modeling sequential data and capturing the temporal dependencies. For instance, Li et al [LMD18] developed a deep neural network system that translates MIDI note data and metric structures into a real‐time skeleton sequence of a pianist playing a keyboard instrument. Their approach combined Convolutional Neural Networks (CNNs) and LSTMs to generate human‐like piano performances.…”