Design Study Methodology: Reflections from the Trenches and the Stacks

Abstract

Abstract-Design studies are an increasingly popular form of problem-driven visualization research, yet there is little guidance available about how to do them effectively. In this paper we reflect on our combined experience of conducting twenty-one design studies, as well as reading and reviewing many more, and on an extensive literature review of other field work methods and methodologies. Based on this foundation we provide definitions, propose a methodological framework, and provide practical guidance for conducting design studies. We define a design study as a project in which visualization researchers analyze a specific real-world problem faced by domain experts, design a visualization system that supports solving this problem, validate the design, and reflect about lessons learned in order to refine visualization design guidelines. We characterize two axes-a task clarity axis from fuzzy to crisp and an information location axis from the domain expert's head to the computer-and use these axes to reason about design study contributions, their suitability, and uniqueness from other approaches. The proposed methodological framework consists of 9 stages: learn, winnow, cast, discover, design, implement, deploy, reflect, and write. For each stage we provide practical guidance and outline potential pitfalls. We also conducted an extensive literature survey of related methodological approaches that involve a significant amount of qualitative field work, and compare design study methodology to that of ethnography, grounded theory, and action research.

Visual enhancement of service maps in logistics clouds

“…A comprehensive design study methodology is proposed in [10] with different phases from preliminary general preconditions (e.g. general design science knowledge acquirement, casting of all stakeholders) over core activities (e.g.…”

“…Afterwards those solutions are to be filtered down to a narrow proposed selection. Further [10] advise to refer on generally accepted design principles and guidelines. The major points are briefly introduced in the following paragraphs.…”

“…As summarized in the section concerning information visualization, the paper follows the methodology of [10] and focuses on developing alternatives, which are evaluated with reference to the main requirements 'usability' and 'scalability' [8]. As categories are taken into account for service retrieval and each category follows a particular type of scale, the following four scales of measurement [25] are considered.…”

Graph Drawing through the Lens of a Framework for Analyzing Visualization Methods

“…This sort of problem-driven work, often called design studies in the visualization literature has rich and interesting challenges, many of which are quite different than those that arise from technique-driven work. A detailed discussion of these issues appears elsewhere, in a recent paper on the methodology of design studies [17], and is beyond the scope of this talk.…”

Understanding Interfirm Relationships in Business Ecosystems with Interactive Visualization

“…Thus, we performed a significant redesign of the system. Following [41] we report on our development process and the new system in this design study.…”