Algorithmic Management for Improving Collective Productivity in Crowdsourcing

Abstract

Crowdsourcing systems are complex not only because of the huge number of potential strategies for assigning workers to tasks, but also due to the dynamic characteristics associated with workers. Maximizing social welfare in such situations is known to be NP-hard. To address these fundamental challenges, we propose the surprise-minimization-value-maximization (SMVM) approach. By analysing typical crowdsourcing system dynamics, we established a simple and novel worker desirability index (WDI) jointly considering the effect of each worker’s reputation, workload and motivation to work on collective productivity. Through evaluating workers’ WDI values, SMVM influences individual workers in real time about courses of action which can benefit the workers and lead to high collective productivity. Solutions can be produced in polynomial time and are proven to be asymptotically bounded by a theoretical optimal solution. High resolution simulations based on a real-world dataset demonstrate that SMVM significantly outperforms state-of-the-art approaches. A large-scale 3-year empirical study involving 1,144 participants in over 9,000 sessions shows that SMVM outperforms human task delegation decisions over 80% of the time under common workload conditions. The approach and results can help engineer highly scalable data-driven algorithmic management decision support systems for crowdsourcing.

Towards AI-powered personalization in MOOC learning

“…Stochastic optimization maximizes or minimizes objective functions or constraints which involve random variables. In the domain of crowdsourcing, data-driven algorithmic management approaches based on stochastic optimization have been used to dynamically divide work among workers so as to fairly allocation workload and achieve superlinear collective productivity, [30][31][32][33][34] schedule restbreaks 35 and adjust incentives to encourage participation based on game theory. 36,37 If MOOC platforms could incorporate mechanisms allowing learners who have interacted with a community TA to provide ratings on the TA's performance, these approaches could be applied in the context of MOOCs to scale up the involvement of community TAs and make more efficient use of community TAs' collective effort.…”

Building Ethics into Artificial Intelligence

“…Interactions [Anderson and Anderson, 2014] [ Dehghani et al, 2008] [Singh, 2014;Singh, 2015] [Battaglino and Damiano, 2015] [Bonnefon et al, 2016] [Blass and Forbus, 2015] [Pagallo, 2016] [Stock et al, 2016 [Sharif et al, 2017] [van Riemsdijk et al, 2015] [ Greene et al, 2016] [Luckin, 2017] [Cointe et al, 2016] [ Noothigattu et al, 2018] [ Yu et al, 2017b] [Conitzer et al, 2017] [Berreby et al, 2017] [Loreggia et al, 2018 [ Wu and Lin, 2018] abling the AI research community to understand human preferences on various ethical dilemmas; 2. Individual Ethical Decision Frameworks: generalizable decision-making mechanisms enabling individual agents to judge the ethics of its own actions and the actions of other agents under given contexts; 3.…”

“…In AI applications which attempt to influence people's behaviours, the principles established by the Belmont Report [Bel, 1978] for behavioural sciences have been suggested to be a starting point for ensuring ethics [Luckin, 2017;Yu et al, 2017b]. The principles include three key requirements: 1) people's personal autonomy should not be violated (they should be able to maintain their free will when interacting with the technology); 2) benefits brought about by the technology should outweigh risks; and 3) the benefits and risks should be distributed fairly among the users (people should not be discriminated based on their personal backgrounds such as race, gender and religion).…”

“…The challenge of measuring benefits and risks remains open for application designers albeit the Ethically Aligned Design guidelines from the IEEE can be a useful starting point [IEEE, 2018]. Computational formulations of human centric values (e.g., collective wellbeing and work-life balance) have been proposed and incorporated into the objective functions of recent AI-powered algorithmic management approaches in crowdsourcing [Yu et al, 2016;Yu et al, 2017a;Yu et al, 2017c].…”

Ethically Aligned Opportunistic Scheduling for Productive Laziness

“…Some direct approaches seek to balance the division of labour among workers in a situationaware manner through data-driven deliberation (Mason and Watts 2012;Yu et al 2013a;Dai et al 2013;Tran-Thanh et al 2014b;Yu et al 2015;Grossmann, Brienza, and Bobocel 2017;Yu et al 2017a). Others design reputation and/or incentive mechanisms to motivate workers to work harder Mao et al 2013;Yu et al 2013b;Tran-Thanh et al 2014a;Zeng, Tang, and Wang 2017).…”