reVISit: Looking Under the Hood of Interactive Visualization Studies

Abstract

Quantifying user performance with metrics such as time and accuracy does not show the whole picture when researchers evaluate complex, interactive visualization tools. In such systems, performance is often influenced by different analysis strategies that statistical analysis methods cannot account for. To remedy this lack of nuance, we propose a novel analysis methodology for evaluating complex interactive visualizations at scale. We implement our analysis methods in reVISit, which enables analysts to explore participant interaction performance metrics and responses in the context of users' analysis strategies. Replays of participant sessions can aid in identifying usability problems during pilot studies and make individual analysis processes salient. To demonstrate the applicability of reVISit to visualization studies, we analyze participant data from two published crowdsourced studies. Our findings show that reVISit can be used to reveal and describe novel interaction patterns, to analyze performance differences between different analysis strategies, and to validate or challenge design decisions.

Predicting intent behind selections in scatterplot visualizations

“…The original study had 128 participants and the follow-up outlier study had 130 participants. After reviewing the provenance data using the reVISit system, 65 we realized that participants sometimes chose not to use predictions in the computer-supported condition. Since our goal was to measure the effects of using predictions, we removed trials that were not completed using predictions in the computer-supported mode.…”

The Role of Expertise on Insight Generation from Visualization Sequences

“…More recently, Cao et al [9] designed a system that allowed the user to label visualizations with personal preferences; then, the system would generate a sequence of visualizations by predicting user preferences based on those labels. User interactions with these visualization systems can be captured using reVISit, a platform that logs navigation behaviors for usability studies [20]. While these approaches are innovative in their own right, they shed little light on how expertise may influence effectiveness of visualization types of sequences.…”

“…3) Visualization and Insight Study Design: Classic studies of cognitive fit theory and visualizations typically were created with benchmark tasks where the insight types and insights themselves were pre-determined by the experimenter [e.g., [20]]. Such classic studies often reported better performance in terms of accuracy or reaction time when the problem was presented in a visual format that fit the specific task.…”

“…These benchmark tasks, also commonly used in user studies of visualizations [20], [25], allow straightforward quantitative analyses, but they additionally prescribe results and threaten ecological validity because realistic data validation scenarios are more open-ended exploration and rarely presented as predetermined tasks [8]. They are also frequently tested on a single visualization rather than sequences of visuals.…”