Intra-list similarity and human diversity perceptions of recommendations: the details matter

Abstract

The diversity of the generated item suggestions can be an important quality factor of a recommender system. In offline experiments, diversity is commonly assessed with the help of the intra-list similarity (ILS) measure, which is defined as the average pairwise similarity of the items in a list. The similarity of each pair of items is often determined based on domain-specific meta-data, e.g., movie genres. While this approach is common in the literature, it in most cases remains open if a particular implementation of the ILS measure is actually a valid proxy for the human diversity perception in a given application. With this work, we address this research gap and investigate the correlation of different ILS implementations with human perceptions in the domains of movie and recipe recommendation. We conducted several user studies involving over 500 participants. Our results indicate that the particularities of the ILS metric implementation matter. While we found that the ILS metric can be a good proxy for human perceptions, it turns out that it is important to individually validate the used ILS metric implementation for a given application. On a more general level, our work points to a certain level of oversimplification in recommender systems research when it comes to the design of computational proxies for human quality perceptions and thus calls for more research regarding the validation of the corresponding metrics.

Real-Time Ideation Analyzer and Information Recommender

“…This is especially adverse in ideation, as this would result in idea echo chambers in which the same ideas would always be confirmed by the recommendations rather than being inspired to conduct critical thinking and generate novel ideas and solutions. To identify the potential for echo chambers, we analyze the intra-list similarity of the recommendations [49,50]. By calculating the cosine similarity between the recommended items, we can determine if an echo chamber exists and if the recommended items only confirm a particular idea.…”

Reviews in Recommender Systems: 2022

“…Answering such questions requires corresponding user studies to, e.g., validate that the computational metrics are good proxies for human perceptions. An attempt to investigate the relationship between perceived diversity and the widely used intra-list similarity measure can be found in Jesse et al (2022).…”

A survey on multi-objective recommender systems

“…Answering such questions requires corresponding user studies to, e.g., validate that the computational metrics are good proxies for human perceptions. An attempt to investigate the relationship between perceived diversity and the widely used intra-list similarity measure can be found in Jesse et al ( 2022 ).…”