Comparing online and lab methods in a problem-solving experiment

Abstract

428The Internet has revolutionized the way in which people communicate and retrieve information. Powerful communication tools are transforming many scientific disciplines, including experimental and clinical psychology. The Web allows access to much wider populations-as well as to populations that were previously difficult to reach-in an inexpensive, fast, and convenient way. In clinical psychology, for example, psychological testing and assessment can be done online (see, e.g., Buchanan, 2002).In experimental psychology, the Web is increasingly being used as an alternative to traditional lab settings for running experiments. In the present article, we review the major advantages and disadvantages of online experiments in comparison with traditional lab experiments; then we compare online versus lab results in problem solving-an area in which this comparison has received less attention. Online Experiment Pros and ConsThere are many potential advantages for doing an experiment online as opposed to in the lab (see Birnbaum, 2004, for a review of pros and cons). First, experimental procedures can be automated, thus reducing costs and the amount of time spent managing the experiment (Reips, 2002a). This also increases the uniformity of the procedure across participants and may reduce demand characteristics (Reips, 2002a). Second, online experiments can be done in a wider array of settings-not just in the highly constrained setting of the lab (Reips, 2000)-and can include 24-h access (Reips, 2002a), considerations that can increase participants' comfort (Salgado & Moscoso, 2003). Third, ethical standards can be maintained because the experiment is publicly available for criticism and the possibility for the coercion of participants is reduced (Reips, 2002a). Finally, online accessibility allows the targeting of specific audiences (through mailing lists or newsgroups) and broadens the participant pool to Web users, rather than, for example, undergraduate students at a particular university, which may allow increased generalizability of the results (Reips, 2000).There are also disadvantages to running an experiment online rather than in the lab. First, the environments will be more variable, including noise, lighting, and technical aspects of the equipment. Effects of this variability may be reduced by asking participants to do the study in a particular sort of environment and by checking for statistical outliers. Second, online experiments are vulnerable to multiple submissions. This seems to be generally rare (Reips, 2000), but it may be more likely when participants have strong opinions about the topic (see, e.g., Konstan, Rosser, Ross, Stanton, & Edwards, 2005). The risk of multiple submissions can be reduced by asking for personal information, using password protection or an IP address verification (Reips, 2002b), and by reducing external incentives, such as winning money or a prize. Finally, there may be biases in the final sample: Only interested and motivated participants may start (self-selection) and complete...

Home is Where the Lab is: A Comparison of Online and Lab Data From a Time-sensitive Study of Interruption

“…A number of studies have been conducted that compare the quality of data collected online with data collected through traditional lab procedures. Dandurand et al (2008) compared online and lab-collected data from a problem-solving experiment. Although online participants were less accurate in their study, the results were considered to be practically equivalent to those gained through lab experimentation.…”

“…As many experimental tasks are performed on computers, researchers have decided to administer studies using the Internet instead of having participants come to a lab (e.g., Snow et al, 2008;Suri & Watts, 2011). Although there are obvious concerns over diminished experimental control when experiments are run online, systematic efforts to compare data suggest that some kinds of HCI research can reliably be conducted with remote online participants (Dandurand, Shultz, & Onishi, 2008;Heer & Bostock, 2010;Komarov, Reinecke, & Gajos, 2013). However, these studies have tended to focus on simple tasks that can be completed relatively quickly.…”

“…We are concerned with how differences between crowds and traditional populations (i.e., crowds as a phenomenon) can affect the results of studies that seek to understand human behaviour. We situate this work in the space of comparative online-offline studies (e.g., Dandurand et al, 2008;Germine et al, 2012;Komarov et al, 2013).…”

Exploring the effects of non-monetary reimbursement for participants in HCI research

“…However, both experiments suffered from high dropout rates, which is a common problem in online experimentation (Dandurand, Shultz, & Onishi, 2008). The dropout rates were higher in the QS experiment (88.1%) than they were in the CitSci experiment (74.57%) and the total number starting the QS experiment was over twice that starting the CitSci experiment (395 v 173 respectively).…”

“…We acknowledge that further research into this will be necessary to make stronger conclusions about this effect, although previous research has suggested differences in performance between labbased and online participants (Dandurand et al, 2008).…”