Grounded evaluation of information visualizations

Isenberg, Petra; Zuk, Torre; Collins, Christopher; Carpendale, Sheelagh

doi:10.1145/1377966.1377974

Cited by 93 publications

(102 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sedlmair et al provide practical advice of how to validate visualizations in large company settings, one of many settings in which a design study may be conducted [76]. Finally, many proposed evaluation methods address the specific needs of validating the usefulness of visualization tools such as the multidimensional in-depth long-term case study approach [80], the insight-based method [68,69], and grounded evaluation [33].…”

Section: Related Workmentioning

confidence: 99%

“…The visualization literature contains a multitude of proposed methods for evaluating visualization tools in the wild [13,33,69,85,95], as well as guidance on using them [39,50]. The most common form of validation are case studies with real users, real problems, and real data, as featured in many strong design studies by others [35,58], and many of our own [10,40,43,44,45,46,71,74,76,77].…”

Section: Deploy: Release and Gather Feedbackmentioning

confidence: 99%

“…The relevant literature instead focuses on methods for designing [1,42,66,79,82,90,91] and evaluating [13,33,39,50,68,69,76,80,85,86,95] visualization tools. We distinguish between methods and methodology with the analogy of cooking; methods are like ingredients, whereas methodology is like a recipe.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Design Study Methodology: Reflections from the Trenches and the Stacks

Sedlmair¹,

Meyer

Munzner³

2012

IEEE Trans. Visual. Comput. Graphics

631

694

View full text Add to dashboard Cite

Abstract-Design studies are an increasingly popular form of problem-driven visualization research, yet there is little guidance available about how to do them effectively. In this paper we reflect on our combined experience of conducting twenty-one design studies, as well as reading and reviewing many more, and on an extensive literature review of other field work methods and methodologies. Based on this foundation we provide definitions, propose a methodological framework, and provide practical guidance for conducting design studies. We define a design study as a project in which visualization researchers analyze a specific real-world problem faced by domain experts, design a visualization system that supports solving this problem, validate the design, and reflect about lessons learned in order to refine visualization design guidelines. We characterize two axes-a task clarity axis from fuzzy to crisp and an information location axis from the domain expert's head to the computer-and use these axes to reason about design study contributions, their suitability, and uniqueness from other approaches. The proposed methodological framework consists of 9 stages: learn, winnow, cast, discover, design, implement, deploy, reflect, and write. For each stage we provide practical guidance and outline potential pitfalls. We also conducted an extensive literature survey of related methodological approaches that involve a significant amount of qualitative field work, and compare design study methodology to that of ethnography, grounded theory, and action research.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Deploy: Release and Gather Feedbackmentioning

confidence: 99%

See 1 more Smart Citation

Design Study Methodology: Reflections from the Trenches and the Stacks

Sedlmair¹,

Meyer

Munzner³

2012

IEEE Trans. Visual. Comput. Graphics

631

694

View full text Add to dashboard Cite

show abstract

“…Qualitative methods can be very valuable for understanding how and why visualizations are used in realistic and meaningful contexts [82]. Qualitative methods, such as those that apply grounded theory, can provide a useful and holistic analysis of visual analytics applications [53]. Since qualitative data are typically collected through direct observation, interviews, and talk-aloud protocols, they are well-suited to lab studies.…”

Section: Measure Typesmentioning

confidence: 99%

Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments

Archambault

Purchase

Hoßfeld

2017

Lecture Notes in Computer Science

View full text Add to dashboard Cite

“…An alternative approach could be to focus on single challenges and aim for a toolbox instead, though this does not acknowledge the interconnectedness of such concepts. Also, our use of a set list of criteria can have limited our evaluation of the concepts by using a single frame on a wide variety of ideas rather than grounded theory (Isenberg et al 2008). …”

Section: Untapped Potential: Collaboration Between Social-ecological mentioning

confidence: 99%

A sense of change: media designers and artists communicating about complexity in social-ecological systems

Vervoort¹,

Keuskamp²,

Kok³

et al. 2014

E&S

View full text Add to dashboard Cite

ABSTRACT. To take on the current and future challenges of global environmental change, fostering a widespread societal understanding of and engagement with the complex dynamics that characterize interacting human and natural systems is essential. Current science communication methods struggle with a number of specific challenges associated with communicating about complex systems. In this study we report on two collaborative processes, a short workshop and longer course, that aimed to harness the insights of interactive media designers and artists to overcome these challenges. The two processes resulted in 86 new interactive media concepts which were selected by the participants and organizers using set criteria and then evaluated using the same criteria by a panel of communication and media design experts and a panel of complex systems scientists using the same criteria. The top eight concepts are discussed in this paper. These concepts fell into the categories of serious games, group interaction concepts, and social media storytelling. The serious games focused directly on complex systems characteristics and were evaluated to be intuitive and engaging designs that combined transparency and complexity well. The group interaction concepts focused mostly on feedbacks and nonlinearity but were fully developed and tested in the workshops, and evaluated as engaging, accessible, and easy to implement in workshops and educational settings. The social media storytelling concepts involved less direct interactions with system dynamics but were seen as highly accessible to large scale audiences. The results of this study show the potential of interdisciplinary collaboration between complex systems scientists, designers, and artists. The results and process discussed in this paper show the value of more structural engagement of interactive media designers and artist communities in the development of communication tools about human and natural systems change.

show abstract

Grounded evaluation of information visualizations

Cited by 93 publications

References 33 publications

Design Study Methodology: Reflections from the Trenches and the Stacks

Design Study Methodology: Reflections from the Trenches and the Stacks

Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments

A sense of change: media designers and artists communicating about complexity in social-ecological systems

Contact Info

Product

Resources

About