Estimating Conversational Styles in Conversational Microtask Crowdsourcing

Abstract

Crowdsourcing marketplaces have provided a large number of opportunities for online workers to earn a living. To improve satisfaction and engagement of such workers, who are vital for the sustainability of the marketplaces, recent works have used conversational interfaces to support the execution of a variety of crowdsourcing tasks. The rationale behind using conversational interfaces stems from the potential engagement that conversation can stimulate. Prior works in psychology have also shown that conversational styles can play an important role in communication. There are unexplored opportunities to estimate a worker's conversational style with an end goal of improving worker satisfaction, engagement and quality. Addressing this knowledge gap, we investigate the role of conversational styles in conversational microtask crowdsourcing. To this end, we design a conversational interface which supports task execution, and we propose methods to estimate the conversational style of a worker. Our experimental setup was designed to empirically observe how conversational styles of workers relate with quality-related outcomes. Results show that even a naive supervised classifier can predict the conversation style with high accuracy (80%), and crowd workers with an Involvement conversational style provided a significantly higher output quality, exhibited a higher user engagement and perceived less cognitive task load in comparison to their counterparts. Our findings have important implications on task design with respect to improving worker performance and their engagement in microtask crowdsourcing.

TickTalkTurk

“…Our experimental findings revealed that workers with specific conversational styles have significantly higher output quality, higher user engagement and less cognitive task load while they are completing a difficult task, and have less task execution time in general. The findings have important implications on worker performance prediction and quality-aware task scheduling in microtask crowdsourcing [3].…”

“…This demo is presented for the system described in our previous work [3]. The code is available online for the benefit of the community (https://github.com/qiusihang/ticktalkturk).…”

DECI: A Tutorial on Designing Effective Conversational Interfaces

“…We will compare conversational crowdsourcing with the traditional web-based microtask crowdsourcing to explain the advantages of conversational crowdsourcing in terms of increasing user satisfaction, improving user engagement, and decreasing perceived workload [15,20]. Next, we will explain the effect of using different conversational styles [27,28], and share empirical insights into how we can define a conversational style, how to estimate the conversational style, and how to exploit the conversational style to facilitate an effective task design [19]. Finally, we will showcase conversational crowdsourcing in a variety of domains, such as supporting microtask execution and aiding informational web search [22].…”

Trainbot: A Conversational Interface to Train Crowd Workers for Delivering On-Demand Therapy

“…It was found that crowd workers exhibited an overall satisfaction while working with the chatbot, and the results produced were of a better quality compared to the web interface (Mavridis et al 2019). Others studied the impact of different conversational styles employed in a text-based conversational agent on the worker performance and engagement, and proposed models to estimate the conversational styles of workers (Qiu, Gadiraju, and Bozzon 2020a;2020b). Results indicated that conversational agents with different conversational styles did not impact the output quality, but they had positive effects on worker engagement and worker retention.…”