A Case for Reproducibility in MIR: Replication of ‘A Highly Robust Audio Fingerprinting System’

Abstract

Claims made in many Music Information Retrieval (MIR) publications are hard to verify due to the fact that (i) often only a textual description is made available and code remains unpublished-leaving many implementation issues uncovered; (ii) copyrights on music limit the sharing of datasets; and (iii) incentives to put effort into reproducible research-publishing and documenting code and specifics on data-is lacking. In this article the problems around reproducibility are illustrated by replicating an MIR work. The system and evaluation described in 'A Highly Robust Audio Fingerprinting System' is replicated as closely as possible. The replication is done with several goals in mind: to describe difficulties in replicating the work and subsequently reflect on guidelines around reproducible research. Added contributions are the verification of the reported work, a publicly available implementation and an evaluation method that is reproducible.

Accuracy comparisons of fingerprint based song recognition approaches using very high granularity

“…Fingerprint generation approaches in song recognition systems can be divided into three different types [33]: the first one describes the energy differences between adjacent frequency bands [15]; the second one locates spectral peaks, using either the relationship with other peaks [23,24,27,29] or the energy information around the peaks to form a fingerprint [1]; the last one uses image retrieval techniques [3,32].…”

“…For comparison purposes, on the MTG-Jamendo subset 00, we used as baselines: 1) Audfprint [11] (using two different parameter settings, denoted as A1 and A2 from here on, where in this latter we modified the "density" parameter in order to obtain the same number of linekeys per second as in our approach); 2) Dejavu [10] (labelled as D); 3) the algorithms implemented in the last released version of Panako [24] (i.e., Olaf labelled as O and Panako, labelled as P). We kept all the parameters of baselines algorithms at the implementation defaults except those specified in Table 1 in such a way the reject option is turned off.…”

“…The second one is Dejavu, 2 another open-source implementation of the Shazamlike approach [29] that uses the constellation algorithm [10]. The third and fourth ones are respectively a new implementation of the classic Shazam algorithm [29], named Olaf, and an updated version of the algorithm described in [24], named Panako, which uses the Gabor transform to move from time domain to spectral domain; Panako and Olaf are distributed as an open-source software. 3 The last one is the algorithm, based on a Philips-like approach, used by the company we are collaborating with for its business.…”

Editorial: Introducing the Transactions of the International Society for Music Information Retrieval

“…In "A Case for Reproducibility in MIR: Replication of 'A Highly Robust Audio Fingerprinting System'", Joren Six, Federica Bressan and Marc Leman address the difficulty of reproducing previous MIR research due to the unavailability of code and music files, often due to copyright issues (Six et al, 2018). The authors illustrate this problem by describing their attempts to replicate a widely cited audio fingerprinting system as closely as possible, generating a reproducible version of the method, and reflecting on guidelines relevant for reproducible algorithms and evaluations.…”

Metric Learning with Sequence-to-sequence Autoencoder for Content-based Music Identification

“…Various techniques have been explored for audio fingerprinting by researchers [1,[4][5][6][7][8], including hand-crafted rule-based algorithms, computer vision-based algorithms, and, more recently, data-driven algorithms. Data-driven algorithms incorporate machine learning techniques for fingerprint generation, with Convolutional Neural Networks (CNN) being the most prominent approach [7,8].…”