Discriminability Tests for Visualization Effectiveness and Scalability

Abstract

The scalability of a particular visualization approach is limited by the ability for people to discern differences between plots made with different datasets. Ideally, when the data changes, the visualization changes in perceptible ways. This relation breaks down when there is a mismatch between the encoding and the character of the dataset being viewed. Unfortunately, visualizations are often designed and evaluated without fully exploring how they will respond to a wide variety of datasets. We explore the use of an image similarity measure, the Multi-Scale Structural Similarity Index (MS-SSIM), for testing the discriminability of a data visualization across a variety of datasets. MS-SSIM is able to capture the similarity of two visualizations across multiple scales, including low level granular changes and high level patterns. Significant data changes that are not captured by the MS-SSIM indicate visualizations of low discriminability and effectiveness. The measure's utility is demonstrated with two empirical studies. In the first, we compare human similarity judgments and MS-SSIM scores for a collection of scatterplots. In the second, we compute the discriminability values for a set of basic visualizations and compare them with empirical measurements of effectiveness. In both cases, the analyses show that the computational measure is able to approximate empirical results. Our approach can be used to rank competing encodings on their discriminability and to aid in selecting visualizations for a particular type of data distribution.

DSSIM: a structural similarity index for floating-point data

“…Also note that the SSIM is quite sensitive to accessories on the plot such as plot grids, labels, etc. (e.g., [19]), which should be avoided. Example of the types of pseudocolor plots of interest are given later in this section (and also in [18]).…”

“…The colormap changes the SSIM less (but still notably for PRECT) when it is similar to the original (Figure 3f). Note that while the SSIM is not influenced by the color or hue of an image [2], [19], [27], when we encode the floatingpoint data into a colormap, the characteristics of the colormap can influence the SSIM [19]. For example, because the prism color map is more segmented than the default, this affects the SSIM more [19] than the cool colormap which is more similar to the default coolwarm map.…”

“…Note that while the SSIM is not influenced by the color or hue of an image [2], [19], [27], when we encode the floatingpoint data into a colormap, the characteristics of the colormap can influence the SSIM [19]. For example, because the prism color map is more segmented than the default, this affects the SSIM more [19] than the cool colormap which is more similar to the default coolwarm map. Other factors that we have not yet mentioned that have been shown to affect the SSIM include the size of the local window, whether or not to use Gaussian weights, and how the edge pixels are treated [28].…”

An Automated Approach to Reasoning About Task-Oriented Insights in Responsive Visualization

“…Prior work has attempted to enable multi-scale views through perceptual organization analysis of a information graphic at each scale [72,73] and hybrid-image visualization that displays different aggregation levels at different viewing distances [35], for example. Signal processing approaches have also been applied to improve the effectiveness of a visualization, for instance, by measuring the difference between the visual salience of a representation and salience of signals in data [37,46], comparing kernel density estimations between a LOESS curve and different representations [71], and extending a structural similarity index for image compression to data visualization [68]. Signal processing-based approaches have typically been applied to single views, and are generally confined to a predefined set of marks and visual variables (e.g., a line chart, a scatterplot), restricting their applicability for settings like ours.…”

Surfacing Visualization Mirages

“…Visualizations where graphical inference is unreliable suggest that either the statistical pattern of interest is not robust or that the visualization design employed is insensitive to such patterns. Proposed mixed-initiative solutions to issues of robustness involve supplementing visualizations with additional metrics that indicate their reliability [8,119,134], or performing pre-analyses to automatically detect potential concerns in a dataset [45]. Lunzer et al [72] explore the robustness of a visualization by superimposing alternative chart configurations.…”