Evaluation Strategies for HCI Toolkit Research

Abstract

Toolkit research plays an important role in the field of HCI, as it can heavily influence both the design and implementation of interactive systems. For publication, the HCI community typically expects toolkit research to include an evaluation component. The problem is that toolkit evaluation is challenging, as it is often unclear what 'evaluating' a toolkit means and what methods are appropriate. To address this problem, we analyzed 68 published toolkit papers. From our analysis, we provide an overview of, reflection on, and discussion of evaluation methods for toolkit contributions. We identify and discuss the value of four toolkit evaluation strategies, including the associated techniques that each employs. We offer a categorization of evaluation strategies for toolkit researchers, along with a discussion of the value, potential limitations, and trade-offs associated with each strategy.

Blossom

“…While there have been many toolkits created for research in human-computer interaction (HCI) [47], there have been few that are specifically designed for constructing robots. Phybots is a toolkit that enables non-roboticists to quickly prototype and program a custom mobile robot [40], but is not designed for social interaction.…”

“…Evaluating toolkits in HCI research is a challenge despite the appreciation for the long-term impact that toolkits have had on HCI research. Ledo et al surveyed evaluation techniques for HCI toolkits over two decades [47] and found that demonstrations and and observation of usage are the most commonly used practices. To evaluate Blossom vis-a-vis its design objectives, we have deployed the system in four demonstration and usage contexts and observed users' interaction with the system.…”

Videostrates

“…We view Videostrates as an example of what Olsen calls user interface systems research [41]. We evaluate it by demonstrating its potential [33] according to Olsen's criteria, as relevant.…”

Complying with Privacy Legislation: From Legal Text to Implementation of Privacy-Aware Location-Based Services

“…The LBS engine is a toolkit in the sense of [36], as it provides a template which helps design location-based services. Asking participants to create example apps within about three months is a strategy for toolkit evaluation and is generally called a take-home study [37]. This approach is useful to gather some evidence on the external validity of the used tools (LBS engine and guidelines).…”