Amazon's Mechanical Turk is an online labor market where requesters post jobs and workers choose which jobs to do for pay. The central purpose of this article is to demonstrate how to use this Web site for conducting behavioral research and to lower the barrier to entry for researchers who could benefit from this platform. We describe general techniques that apply to a variety of types of research and experiments across disciplines. We begin by discussing some of the advantages of doing experiments on Mechanical Turk, such as easy access to a large, stable, and diverse subject pool, the low cost of doing experiments, and faster iteration between developing theory and executing experiments. While other methods of conducting behavioral research may be comparable to or even better than Mechanical Turk on one or more of the axes outlined above, we will show that when taken as a whole Mechanical Turk can be a useful tool for many researchers. We will discuss how the behavior of workers compares with that of experts and laboratory subjects. Then we will illustrate the mechanics of putting a task on Mechanical Turk, including recruiting subjects, executing the task, and reviewing the work that was submitted. We also provide solutions to common problems that a researcher might face when executing their research on this platform, including techniques for conducting synchronous experiments, methods for ensuring high-quality work, how to keep data private, and how to maintain code security.

No abstract

Complex problems in science, business, and engineering typically require some tradeoff between exploitation of known solutions and exploration for novel ones, where, in many cases, information about known solutions can also disseminate among individual problem solvers through formal or informal networks. Prior research on complex problem solving by collectives has found the counterintuitive result that inefficient networks, meaning networks that disseminate information relatively slowly, can perform better than efficient networks for problems that require extended exploration. In this paper, we report on a series of 256 Web-based experiments in which groups of 16 individuals collectively solved a complex problem and shared information through different communication networks. As expected, we found that collective exploration improved average success over independent exploration because good solutions could diffuse through the network. In contrast to prior work, however, we found that efficient networks outperformed inefficient networks, even in a problem space with qualitative properties thought to favor inefficient networks. We explain this result in terms of individual-level explore-exploit decisions, which we find were influenced by the network structure as well as by strategic considerations and the relative payoff between maxima. We conclude by discussing implications for realworld problem solving and possible extensions.M any problems that arise in science, business, and engineering are "complex" in the sense that they require optimization along multiple dimensions, where changes in one dimension can have different effects depending on the values of the other dimensions. A common way to represent problem complexity of this nature is with a "fitness landscape," a multidimensional mapping from some choice of solution parameters to some measure of performance, where complexity is expressed by the "ruggedness" of the landscape (1, 2). A simple problem, that is, would correspond to a relatively smooth landscape in which the optimal solution can be found by strictly local and incremental exploration around the current best solution. By contrast, a complex problem would correspond to a landscape with many potential solutions ("peaks") separated by low-performance "valleys." In the event that the peaks are of varying heights, purely local exploration on a rugged landscape can lead to solutions that are locally optimal but globally suboptimal. The result is that when solving complex problems, problem solvers must strike a balance between local exploitation of already discovered solutions and nonlocal exploration for potential new solutions (3, 4).In many organizational contexts, the tradeoff between exploration and exploitation is affected by the presence of other problem solvers who are attempting to solve the same or similar problems, and who communicate with each other through some network of formal or informal social ties (5-10). Intuitively, it seems clear that communication networks should aid collaborative p...