Warping time for more effective real-time crowdsourcing

Abstract

In this paper, we introduce the idea of "warping time" to improve crowd performance on the difficult task of captioning speech in real-time. Prior work has shown that the crowd can collectively caption speech in real-time by merging the partial results of multiple workers. Because non-expert workers cannot keep up with natural speaking rates, the task is frustrating and prone to errors as workers buffer what they hear to type later. The TimeWarp approach automatically increases and decreases the speed of speech playback systematically across individual workers who caption only the periods played at reduced speed. Studies with 139 remote crowd workers and 24 local participants show that this approach improves median coverage (14.8%), precision (11.2%), and per-word latency (19.1%). Warping time may also help crowds outperform individuals on other difficult real-time performance tasks.

Real-time captioning by non-experts with legion scribe

“…To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. We have also shown that by systematically slowing down and speeding up the audio for individual workers we can improve both precision and recall by more than 10% [2]. This is the TimeWarp approach to real-time human computation.…”

Fostering parent–child dialog through automated discussion suggestions

“…Finally, recent work in crowdsourcing has focused on providing fast response times on tasks (Bernstein et al 2011;Lasecki et al 2013), with focus problems such as quality control (Mashhadi and Capra 2011). Thus, we consider that one key advantage our work has over expert-sourced suggestions is that it provides shorter response times, with a significant part of the suggestion process automated by topic and suggestion generation algorithms.…”

Crowdsourcing Synchronous Spectator Support

“…In the cheering case however, the support has to happen at a specific time and workers have to 'sync' with the event rather then vice versa. Related work is found in studies on crowd-powered interfaces with highly innovative techniques for crowdsourcing just-in-time work such as VizWiz -a system for crowdsourcing near real-time support for vision impaired [7], Lasecki et al's ingenious work for captioning live speech [20] and Bernstein et al's work on queuing workers using multiple queuing models [4]. However, with the exception of Morris et al's work on 'Crowdsourcing Collective Emotional Intelligence' [22], there is very little knowledge on crowdsourcing spectator support.…”