Improving Worker Engagement Through Conversational Microtask Crowdsourcing

Abstract

The rise in popularity of conversational agents has enabled humans to interact with machines more naturally. Recent work has shown that crowd workers in microtask marketplaces can complete a variety of human intelligence tasks (HITs) using conversational interfaces with similar output quality compared to the traditional Web interfaces. In this paper, we investigate the effectiveness of using conversational interfaces to improve worker engagement in microtask crowdsourcing. We designed a text-based conversational agent that assists workers in task execution, and tested the performance of workers when interacting with agents having different conversational styles. We conducted a rigorous experimental study on Amazon Mechanical Turk with 800 unique workers, to explore whether the output quality, worker engagement and the perceived cognitive load of workers can be affected by the conversational agent and its conversational styles. Our results show that conversational interfaces can be effective in engaging workers, and a suitable conversational style has potential to improve worker engagement. Our findings have important implications on workflows and task design with regard to better engaging workers in microtask crowdsourcing marketplaces.

Just the Right Mood for HIT!

“…A conversational style is actually the superimposition of multiple linguistic features and devices [25]. To this end, we selected features and devices that can be applied in our case to create conversation agents emulating High Involvement and High Considerateness conversational styles according to the design criteria from the previous work [23]. Selected features are shown in Table 1.…”

“…Psychologists and linguists have found that conversational styles play an important role in communication [15,24,25]. Our recent study has investigated whether adapting and personalizing the conversational style of an agent to that of a worker can improve the quality of work [23]. We aim to fill this knowledge gap by addressing the following research questions: RQ1: How do worker moods affect their performance, engagement and cognitive load in conversational microtask crowdsourcing?…”

TickTalkTurk

“…We found that a suitable conversational style has the potential to engage workers further in specific task types. Our work takes crucial strides towards furthering the understanding of conversational interfaces for microtasking, revealing insights into the role of conversational styles across a variety of tasks [4].…”

“…A notable feature of the interaction between crowdsourcing platforms and workers in the majority of prior work, is the use of traditional web-based GUIs to communicate with workers, transmit instructions and gather responses thereafter. In our recently introduced notion of conversational microtask crowdsourcing, a conversational agent interfaces online workers with the crowdsourcing platform, facilitating task execution and task completion [1,4].…”

Estimating Conversational Styles in Conversational Microtask Crowdsourcing

“…Previous work studied how specific linguistic strategies and language styles can affect work outcomes [27,46]. In a recent study, we investigated whether different conversational styles of an agent can increase worker engagement [44]. However, to the best of our knowledge, current conversational agents (particularly for crowdsourcing) have only studied the effects of the conversational style of agents, rather than the conversational style of online users (i.e., workers in the context of microtask crowdsourcing).…”