A Comparison of Human, Automatic and Collaborative Music Genre Classification and User Centric Evaluation of Genre Classification Systems

Abstract

Abstract. In this paper two sets of evaluation experiments are conducted. First, we compare state-of-the-art automatic music genre classification algorithms to human performance on the same dataset, via a listening experiment. This will show that the improvements of contentbased systems over the last years have reduced the gap between automatic and human classification performance, but could not yet close this gap. As an important extension to previous work in this context, we will also compare the automatic and human classification performance to a collaborative approach. Second, we propose two evaluation metrics, called user scores, that are based on the votes of the participants of the listening experiment. This user centric evaluation approach allows to get rid of predefined ground truth annotations and allows to account for the ambiguous human perception of musical genre. To take genre ambiguities into account is an important advantage with respect to the evaluation of content-based systems, especially since the dataset compiled in this work (both the audio files and collected votes) are publicly available.

Classification accuracy is not enough

“…The work proposing AdaBFFs (Bergstra et al 2006a), SRCAM (Panagakis et al 2009b), and the features of MAPsCAT (Andén and Mallat 2011), present only classification accuracy. Furthermore, based on classification accuracy, Seyerlehner et al (2010) argue that the performance gap between MGR systems and humans is narrowing; and in this issue, Humphrey et al conclude "progress in content-based music informatics is plateauing" (Humphrey et al 2013). Figure 2 shows that with respect to the classification accuracies in GTZAN reported in 83 published works (Sturm 2013b), those of AdaBFFs, SRCAM, and MAPsCAT lie above what is reported best in half of this work.…”

“…The work by Vatolkin (2012) provides a comparison of various performance statistics for music classification. Other works (Berenzweig et al 2004;Craft et al 2007;Craft 2007;Lippens et al 2004;Wiggins 2009;Seyerlehner et al 2010;Sturm 2012b) argue for measuring performance in ways that take into account the natural ambiguity of music genre and similarity. For instance, we Sturm (2012b), Craft et al (2007) and Craft (2007) argue for richer experimental designs than having a system apply a single label to music with a possibly problematic "ground truth."…”

“…Typically, formally justifying a misclassification as an error is a task research in MGR often defers to the "ground truth" of a dataset, whether created by a listener (Tzanetakis and Cook 2002), the artist (Seyerlehner et al 2010), music vendors (Gjerdingen and Perrott 2008;Ariyaratne and Zhang 2012), the collective agreement of several listeners (Lippens et al 2004;García et al 2007) professional musicologists (Abeßer et al 2012), or multiple tags given by an online community (Law 2011). Table 2 shows the datasets used by references in our survey (Sturm 2012a).…”

“…Genres which are assumed to be very different, like Metal and Classic, were never confused." The human-like confusion tables found in MGR work, as well as the ambiguity between music genres, motivates evaluating MGR systems by considering as less troublesome the confusions we expect from humans Craft 2007;Lippens et al 2004;Seyerlehner et al 2010).…”

“…We show these FoMs-all of which have been used in the past to rank MGR systems, e.g., Chai and Vercoe (2001), Tzanetakis and Cook (2002), Aucouturier and Pachet (2003), Burred and Lerch (2004), Turnbull and Elkan (2005), Flexer (2006), DeCoro et al (2007), Benetos and Kotropoulos (2008), Panagakis et al (2009b), Bergstra et al (2010), Fu et al (2011) and Ren and Jang (2012) citing one work from each year since 2001-do not reliably reflect the capacity of an MGR system to recognize genre. While these claims have not been made overt in any of the 467 references we survey (Sturm 2012a), shades of it have appeared before Craft 2007;Lippens et al 2004;Wiggins 2009;Seyerlehner et al 2010;Sturm 2012b), which argue for evaluating performance in ways that account for the ambiguity of genre being in large part a subjective construction (Fabbri 1982;Frow 2005). We go further and argue that the evaluation of MGR systems-the experimental designs, the datasets, and the FoMs-and indeed, the development of future systems, must embrace the fact that the recognition of genre is to a large extent a musical problem, and must be evaluated as such.…”