Evaluating the effectiveness of explanations for recommender systems

Abstract

When recommender systems present items, these can be accompanied by explanatory information. Such explanations can serve seven aims: effectiveness, satisfaction, transparency, scrutability, trust, persuasiveness, and efficiency. These aims can be incompatible, so any evaluation needs to state which aim is being investigated and use appropriate metrics. This paper focuses particularly on effectiveness (helping users to make good decisions) and its trade-off with satisfaction. It provides an overview of existing work on evaluating effectiveness and the metrics used. It also highlights the limitations of the existing effectiveness metrics, in particular the effects of under-and overestimation and recommendation domain. In addition to this methodological contribution, the paper presents four empirical studies in two domains: movies and cameras. These studies investigate the impact of personalizing simple feature-based explanations on effectiveness and satisfaction. Both approximated and real effectiveness is investigated. Contrary to expectation, personalization was detrimental to effectiveness, though it may improve user satisfaction. The studies also highlighted the importance of considering opt-out rates and the underlying rating distribution when evaluating effectiveness.

Principles of Explanatory Debugging to Personalize Interactive Machine Learning

“…Increased transparency has been associated with many benefits, including increased user acceptance of (or satisfaction with) recommendations and predictions [4,8,39], improved training of intelligent agents [38], and increased trust in the system's predictions [10]. In this paper, however, we primarily care about transparency because it has been shown to help users understand how a learning system operates [23,28].…”

Inspectability and control in social recommenders

“…The reasoning and insight into the recommendation process exposed by an explanation interface can also increase the inspectability of the system as a whole. Tintarev and Masthoff [46] show that explanations make it easier to judge the quality of recommendations. Consequently, such explanations increase users' trust in the recommendations and, in turn, the perceived competence of the system ( [8,11], see also [48]).…”

“…Users seem to appreciate it when recommender systems explain their recommendations [8,11,19,45,46,48]. In social recommenders, where users know the people on which the recommendations are based, the system can provide such explanation by showing how the overlap between one's preferences and those of one's friends resulted in a set of recommendations.…”

Recommender systems: from algorithms to user experience

“…Hook et al (1996 discussed ways to form explanations in adaptive systems. Tintarev and Masthoff (2012) provide a more detailed review of research on explanations in recommender systems. We share their conclusion that additional work is needed in the field, particularly to explore the extent to which the recommendations are actually improving user decision-making, to explore the costs and benefits of scrutability in improving recommendations, and to understand the effects of explanations in the context of live, deployed systems.…”