Assessing the Effect of Visualizations on Bayesian Reasoning through Crowdsourcing

Abstract

People have difficulty understanding statistical information and are unaware of their wrong judgments, particularly in Bayesian reasoning. Psychology studies suggest that the way Bayesian problems are represented can impact comprehension, but few visual designs have been evaluated and only populations with a specific background have been involved. In this study, a textual and six visual representations for three classic problems were compared using a diverse subject pool through crowdsourcing. Visualizations included area-proportional Euler diagrams, glyph representations, and hybrid diagrams combining both. Our study failed to replicate previous findings in that subjects' accuracy was remarkably lower and visualizations exhibited no measurable benefit. A second experiment confirmed that simply adding a visualization to a textual Bayesian problem is of little help, even when the text refers to the visualization, but suggests that visualizations are more effective when the text is given without numerical values. We discuss our findings and the need for more such experiments to be carried out on heterogeneous populations of non-experts.

Improving Bayesian Reasoning: The Effects of Phrasing, Visualization, and Spatial Ability

“…Combining techniques that have previously been tested independently, our results show an increase in the accuracy of the mammography problem from the previously reported 6% [29] to 42%. Our findings demonstrate how the phrasing of a Bayesian problem can partially explain the poor or inconsistent results of prior work and provide a baseline text-only representation for future work.…”

“…Researchers have also explored visualizations such as decision trees [13,28], contingency tables [7], "beam cut" diagrams [17] and probability curves [7], and have shown improvements over textonly representations. However, when researchers in the visualization community extended this work to a more diverse sampling of the general population, they found that adding visualizations to existing text representations did not significantly increase accuracy [29,32].…”

Visualizing Sets: An Empirical Comparison of Diagram Types

“…The MTurk HITs were based on the templates provided by Micallef et al [24], at http://www.aviz.fr/bayes. Every question, in both the training and the main study, was displayed on a separate page of the HIT.…”

Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments

“…A larger number of participants, first of all, result in larger samples (e.g., 480 participants in [74], 550 in [32]). Having more samples makes the data analysis more robust to outliers, since outliers can be removed while maintaining a large number of "good" samples.…”

“…For example, finding paths in an abstract graph visualization is less likely engaging than finding the friends that connect two people in a social network. Micallef et al [74] report on participants commenting on their interest and engagement in the study, and on things they learned while participating. Section 5 provides a few suggestions on how this could be achieved.…”