Empirical Studies in Information Visualization: Seven Scenarios

Abstract

We take a new, scenario-based look at evaluation in information visualization. Our seven scenarios, evaluating visual data analysis and reasoning, evaluating user performance, evaluating user experience, evaluating environments and work practices, evaluating communication through visualization, evaluating visualization algorithms, and evaluating collaborative data analysis were derived through an extensive literature review of over 800 visualization publications. These scenarios distinguish different study goals and types of research questions and are illustrated through example studies. Through this broad survey and the distillation of these scenarios, we make two contributions. One, we encapsulate the current practices in the information visualization research community and, two, we provide a different approach to reaching decisions about what might be the most effective evaluation of a given information visualization. Scenarios can be used to choose appropriate research questions and goals and the provided examples can be consulted for guidance on how to design one's own study.

Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments

“…Lam et al [58] provide a more comprehensive discussion on visualization evaluation, and we detail four examples of crowdsourcing-based studies in Section 8. Ideally, crowdsourcing technologies should eventually support the design and deployment of studies spanning this space with minimal overhead on the experimenter.…”

Crowdsourcing for Information Visualization: Promises and Pitfalls

“…The type of evaluation that can be performed using crowdsourcing is usually referred to as a participant performance evaluation. Lam et al [58] identify two question types for these studies:…”

A chronographic protocol for modelling construction projects

“…The proposal of a new conceptual framework and study of the suitable visual parameters and their associated values in order to define a standard graphical protocol. The research methodology followed the steps recommended by Lam et al (2012), described below.…”

“…The validation phase evaluated the application of the proposed graphic convention. The case study was conducted in two stages by following two of the seven evaluation scenarios proposed by Lam et al (2012), namely, the VDAR (evaluating visual data analysis and reasoning) and the CTV (evaluating communication through visualisation). The first stage assessed the suitability of the texture convention and its visual clarity and the second stage evaluated the mental effort required to find the desired information on a building construction schedule.…”