How Good is 85%?

Abstract

Many HCI and ubiquitous computing systems are characterized by two important properties: their output is uncertain-it has an associated accuracy that researchers attempt to optimize-and this uncertainty is user-facing-it directly affects the quality of the user experience. Novel classifiers are typically evaluated using measures like the F 1 scorebut given an F-score of (e.g.) 0.85, how do we know whether this performance is good enough? Is this level of uncertainty actually tolerable to users of the intended application-and do people weight precision and recall equally? We set out to develop a survey instrument that can systematically answer such questions. We introduce a new measure, acceptability of accuracy, and show how to predict it based on measures of classifier accuracy. Out tool allows us to systematically select an objective function to optimize during classifier evaluation, but can also offer new insights into how to design feedback for user-facing classification systems (e.g., by combining a seemingly-low-performing classifier with appropriate feedback to make a highly usable system). It also reveals potential issues with the ubiquitous F1-measure as applied to user-facing systems.

Towards a visual guide for communicating uncertainty in Visual Analytics

“…Formally describing this multi-faceted entity may be very difficult and, once fathomed, may still lack expressiveness and meaning [13].…”

“…This in turn directly influences the accuracy of the analysis. Yet, simply indicating the overall model accuracy of such an algorithm in a prediction may often not be enough for the user to make an informed decision [13]. Instead, they should be able to inspect the workings of the employed algorithm to learn how well their data fits the classification space.…”

“…Using a visual representation of uncertainty when visualising an algorithm's mechanics, allows users of Visual Analytics applications to inspect how well their data fits a model. Giving just a numerical estimate of the prediction quality may neither be very expressive [13], nor helpful with improving the analysis. By directly visualising the algorithm mechanics and the classification or prediction accuracy for data points in certain zones, users of these applications can make a much better and much more informed decision.…”

“…This introduces various types of uncertainty, the display of which is not adequately considered in these powerful toolkits used by users not experienced in visual data analysis. Simply giving a numerical quality estimate of a prediction based on such data may not be enough [13]. It is therefore necessary to not only visualise the classification space of an algorithm used for a prediction or visualisation, but also to explore how different types of uncertainty are best represented to such "laymen" and to provide a visual guide, whose recommendations can be implemented into established frameworks to address this omittance.…”

“…Similarly, data points far away from a cluster's centre in a K-means clustering algorithm can be identified and the algorithm adjusted to improve classification accuracy and to reduce "fuzzy" labelling. Therefore, a graphical representation of uncertainty inside the classification space that communicates the "goodness of fit" of various data points to the model, may be much more effective than a numeric indicator of the overall prediction quality [13].…”