Methodological insights from a rigorous small scale design experiment

Abstract

This paper discusses the methods used to conduct high quality small-scale design experiments. It aims to provide a demonstrator promoting the uptake of more rigorous methods in design research and based on this it aims to specify a body of further work for linking study types and contexts. A small-scale experiment was conducted using methods specifically developed to mitigate four core problem areas identified from review: context, system understanding, methods and controls. The techniques were then critiqued in detail and used to draw several insights for design researchers including the value of control techniques and triangulation of metrics. Finally, the critique is used to specify further research aimed at linking design experimentation and design practice more effectively for design research.Design researchers have used experiments and observational studies extensively over the last forty years to explore the working practises and performance of designers and design teams (Cross, 2007). Recent examples includeHoward et al.'s (2010) work on ideation, Dong's (2005) work on analysing design team communication, Bakeman & Deckner's (2003) work on behaviour others across a range of areas (Ball & Ormerod, 2000; M. A. Robinson, et al., 2005). Empirical study forms a valuable part of design research, providing essential insight into many areas of design whilst also supporting theorybuilding (Stempfle & Badke-schaub, 2002) and the development of real world impact as emphasised by Briggs (2006) and Cross et al. (1996) . However, there is an ongoing challenge to improve the quality of empirical studies in design research (Blessing & Chakrabarti, 2009).One standard approach to improving quality has been to develop large-scale statistical studies, however, these are very time/resource intensive and not always appropriate for design research topics as such are rarely used in design research. A common technique that is widely used is small-scale studies. Thus, this paper examines how small-scale studies can be made more rigorous, with the aim of providing a demonstrator to support uptake of underutilised methods in design research. To this end, a laboratorybased design experiment has been developed to explore what methods can be used to improve validity, replicability and reliability. Although these studies are not a substitute for large-scale statistical validation, this paper will show that, with improved rigorous methods, small-scale studies can show possible trends and give insights into design situations. This is demonstrated using a case study experiment with the hypothesis: design teams benefit from having design relevant information presented to them during the early design phases of a product development process. The experiment also aimed to investigate what format would be most effective for the pushed information: video footage of users interacting with the product or numerical data describing the same interactions. Five design teams were each tasked with generating design ideas for a domestic refrigerator with ...

Prototyping with your hands: the many roles of gesture in the communication of design concepts

“…This is in line with some prior gesture studies (Yasui, 2013) and previous surveys of team size effects (Cash, Elias, Dekoninck, & Culley, 2012). In particular dyads have been shown to display different generic communication behaviour to larger groups (Hackman & Vidmar, 1970;Salas et al, 2008) while teams of three have been found to be analogous to teams of up to five people (Cash et al, 2012;Hackman & Vidmar, 1970).…”

A comparison of designer activity using core design situations in the laboratory and practice

“…Twelve students were randomly selected and split into four teams of three. A team size of three was selected as it fulfilled the demands of representativeness (Cash, Elias, et al, 2011), whilst also allowing for effective comparison to the activities of the practitioners in Study 1.…”

The influence of documenting assumed values of product properties on the number of iterations in the design process - first observations

“…While this measure was not expected to have a positive influence, it was used to give the participants the impression of receiving helpful support emulating the effects of what in medical trials is referred to as a placebo (Cash et al, 2012).…”