Lessons Learned from Crowdsourcing Complex Engineering Tasks

Abstract

CrowdsourcingCrowdsourcing is the practice of obtaining needed ideas, services, or content by requesting contributions from a large group of people. Amazon Mechanical Turk is a web marketplace for crowdsourcing microtasks, such as answering surveys and image tagging. We explored the limits of crowdsourcing by using Mechanical Turk for a more complicated task: analysis and creation of wind simulations.Harnessing Crowdworkers for EngineeringOur investigation examined the feasibility of using crowdsourcing for complex, highly technical tasks. This was done to determine if the benefits of crowdsourcing could be harnessed to accurately and effectively contribute to solving complex real world engineering problems. Of course, untrained crowds cannot be used as a mere substitute for trained expertise. Rather, we sought to understand how crowd workers can be used as a large pool of labor for a preliminary analysis of complex data.Virtual Wind TunnelWe compared the skill of the anonymous crowd workers from Amazon Mechanical Turk with that of civil engineering graduate students, making a first pass at analyzing wind simulation data. For the first phase, we posted analysis questions to Amazon crowd workers and to two groups of civil engineering graduate students. A second phase of our experiment instructed crowd workers and students to create simulations on our Virtual Wind Tunnel website to solve a more complex task.ConclusionsWith a sufficiently comprehensive tutorial and compensation similar to typical crowd-sourcing wages, we were able to enlist crowd workers to effectively complete longer, more complex tasks with competence comparable to that of graduate students with more comprehensive, expert-level knowledge. Furthermore, more complex tasks require increased communication with the workers. As tasks become more complex, the employment relationship begins to become more akin to outsourcing than crowdsourcing. Through this investigation, we were able to stretch and explore the limits of crowdsourcing as a tool for solving complex problems.

Microwork platforms as enablers to new ecosystems and business models: the challenge of managing difficult tasks

“…These findings and observations, and the evolution of microwork platforms, are perfectly in line with the recent results of Staffelbach et al (2015), who compare results from AMT workers with those of civil engineering graduate students in the context of the analysis of wind simulation data. They conclude that increased communication was needed for the handling of more complex tasks, and that the employment relationship then began to be closer to outsourcing.…”

A Cyber-Based Data-Enabled Virtual Organization for Wind Load Effects on Civil Infrastructures: VORTEX-Winds

“…Despite initial serious criticism by some with background in computational wind engineering, we are pleased to report that we have measureable success in the initial development and have noticed a step forward in potentially removing the commonly perceived fear out of CFD that has haunted many earlier users . It has been demonstrated that the concept, though initially opposed by some experts, is feasible as a group of student engineers, low-skill volunteers from Amazon Mechanical Turks, graduate students, and postdoctoral graduates were able to simulate flow around a cross-section of the Empire State Building with very reliable results as the users went through helpful tutorials, explored various grid options, turbulence models, and undertook convergence studies (Staffelbach et al, 2015). VWT is a user-friendly web-enabled CFD collaborative platform that does not require expertise of CFD simulation and extensive computer resources.…”

“…Untrained workers can also run simulations by taking advantage of a bank of preliminary analyses of complex data they can import from the platform. Investigations to examine the feasibility of using crowdsourcing for various complexities of technical tasks using VWT have been carried out, and it shows that crowd workers with no background in engineering could follow tutorials and come up with acceptable solutions (Staffelbach et al, 2015). To complete a task on VWT, a user should create the design, generate the mesh, and run the simulation.…”

“…A good agreement was observed as shown both in Figure 12C and Table 2. More examples can be found in Staffelbach et al (2015).…”

Crowdsourcing Analysis of Twitter Data on Climate Change: Paid Workers vs. Volunteers

“…It has been repeatedly demonstrated that complex problems normally requiring advanced technical training can be solved by crowdsourcing; examples include civil engineering [25], bioinformatics [26], astronomy [12] and many others. Furthermore, under certain conditions, even generating novel ideas and innovations can be crowdsourced with results comparable to those obtained from experts [27].…”