A Bayesian Mallows approach to nontransitive pair comparison data: How human are sounds?

Abstract

We are interested in learning how listeners perceive sounds as having human origins. An experiment was performed with a series of electronically synthesized sounds, and listeners were asked to compare them in pairs. We propose a Bayesian probabilistic method to learn individual preferences from non-transitive pairwise comparison data, as happens when one (or more) individual preferences in the data contradicts what is implied by the others. We build a Bayesian Mallows model in order to handle non-transitive data, with a latent layer of uncertainty which captures the generation of preference misreporting. We then develop a mixture extension of the Mallows model, able to learn individual preferences in a heterogeneous population. The results of our analysis of the musicology experiment are of interest to electroacoustic composers and sound designers, and to the audio industry in general, whose aim is to understand how computer generated sounds can be produced in order to sound more human.

BayesMallows: An R Package for the Bayesian Mallows Model

“…The Logistic model is a generalization of (7) where, instead of assigning a constant value θ to the probability of making a mistake, it depends on the distance between the ranks of the two items under comparison. In Crispino et al (2018) the Logistic model gave results very similar to the Bernoulli model, and currently only the Bernoulli model is available in BayesMallows. The sampling scheme is similar to the one used for the case of transitive pairwise preferences, apart from an additional step for updating θ, and the augmentation scheme for Rj , which is slightly different.…”

“…Another frequent situation is preferences collected over time, so that assessors might change opinion along data collection. This setting is considered in Crispino, Arjas, Vitelli, Barrett, and Frigessi (2018), where the model for transitive pairwise comparisons of Section 2.4 is generalized to handle situations where non-transitivities in the data occur. Note that the kind of non-transitivity that is considered in Crispino et al (2018) considers only the individual level preferences.…”

“…This setting is considered in Crispino, Arjas, Vitelli, Barrett, and Frigessi (2018), where the model for transitive pairwise comparisons of Section 2.4 is generalized to handle situations where non-transitivities in the data occur. Note that the kind of non-transitivity that is considered in Crispino et al (2018) considers only the individual level preferences. A different type of non-transitivity, which we do not consider here, arises when aggregating preferences across assessors, as under Condorcet (Marquis of Condorcet 1785) or Borda (de Borda 1781) voting rules.…”

“…Two models for (6) are considered in Crispino et al (2018): the Bernoulli model, which accounts for random mistakes, and the Logistic model, which lets the probability of making a mistake depend on the similarity of the items being compared. The Bernoulli model states that:…”

“…The sampling scheme is similar to the one used for the case of transitive pairwise preferences, apart from an additional step for updating θ, and the augmentation scheme for Rj , which is slightly different. We refer to Crispino et al (2018) for details.…”