CALVI: Critical Thinking Assessment for Literacy in Visualizations

Abstract

Visualization misinformation is a prevalent problem, and combating it requires understanding people’s ability to read, interpret, and reason about erroneous or potentially misleading visualizations, which lacks a reliable measurement: existing visualization literacy tests focus on well-formed visualizations. We systematically develop an assessment for this ability by: (1) developing a precise definition of misleaders (decisions made in the construction of visualizations that can lead to conclusions not supported by the data), (2) constructing initial test items using a design space of misleaders and chart types, (3) trying out the provisional test on 497 participants, and (4) analyzing the test tryout results and refining the items using Item Response Theory, qualitative analysis, a wrong-due-to-misleader score, and the content validity index. Our final bank of 45 items shows high reliability, and we provide item bank usage recommendations for future tests and different use cases. Related materials are available at: https://osf.io/pv67z/.

Reading a graph is like reading a paragraph.

“…These steps are important even for trivially simple visualizations. Finally, if a visualization designer can guide a viewer's capacity-limited relationship processing to the "right" pattern, then graphical literacy training should include monitoring for bad actors who use the same technique to guide people to the "wrong pattern" (Ge et al, 2023), much like a magician might use subtle attentional misdirection to hide an action from an audience (Kuhn et al, 2008).…”

reVISit: Supporting Scalable Evaluation of Interactive Visualizations

“…Likewise, Borkin et al investigated what makes a visualization memorable by presenting participants with a sequence of static images and subsequently asking if they had seen an image before [1]. The trial format also has been adopted by many visualization literacy assessments [3,8,12]. Notably, many recent studies are conducted as online experiments, placing a high technical burden on the experimenters.…”