Investigating confidence displays for top‐N recommendations

Abstract

Recommendation systems often compute fixed length lists of recommended items to users. Forcing the system to predict a fixed length list for each user may result in different confidence levels for the computed recommendations. Reporting the system's confidence in its predictions (the recommendation strength) can provide valuable information to users in making their decisions. In this paper we investigate several different displays of the system's confidence to users and conclude that some displays are easier to understand and are favored by most users. We continue to investigate the effect confidence has on users, in terms of their perception of the recommendation quality and the user experience with the system. Our studies show that it was not easier for users to identify relevant items when confidence is displayed. Still, users appreciated the displays, and trusted them when the relevance of items is difficult to establish.

Does a Confidence Level for Automated Driving Time Estimations Improve the Subjective Evaluation of an Automation HMI?

“…In conclusion, this study adds to previous contradictory results of confidence display studies. In some studies, confidence displays achieved no improvements or even some disadvantages [11,12,17]. Moreover, in today's navigation systems, which suffer the same problem, no confidence is given in the arrival time estimation.…”

“…The visualization of uncertainty information has already been an important topic in aviation [10] and for recommender systems [11]. Moreover, confidence displays gain importance in the automotive industry due to the ongoing development of advanced driving assistance functions.…”

“…Shani et al [11] researched confidence displays for recommender systems for news stories, video clips, shopping items, etc. In a user study, they found some displays to be more understandable and better liked by the users: Concepts inspired by well-known displays in other areas (e.g., the bar chart from phone connectivity or fuel gauge) are favored.…”

“…Based on the preliminary study results, the final concept (see Figure 2) included a bar concept to abstract the confidence level, as the numerical concept was well rated but also considered unrealistically concrete. Furthermore, Shani et al [11] recommended well-known and discrete concepts with a low number of scores. To avoid the symbol being misunderstood as a representation of time passing by, we converted the vertical bar into a horizontal bar resembling the icon for mobile phone connection quality, which is based on the recommendations of pre-study participants and is supported by findings of Shani et al [11].…”

“…Furthermore, Shani et al [11] recommended well-known and discrete concepts with a low number of scores. To avoid the symbol being misunderstood as a representation of time passing by, we converted the vertical bar into a horizontal bar resembling the icon for mobile phone connection quality, which is based on the recommendations of pre-study participants and is supported by findings of Shani et al [11]. Furthermore, the color-coding was abandoned as especially red, but also yellow, convey a criticality that does not fit a confidence level for time budget estimations.…”