Responsible research with crowds

Abstract

High-level guidelines for the treatment of crowdworkers.

The Ethics of AI Ethics: An Evaluation of Guidelines

“…Issues such as lithium mining, e-waste, the one-way use of rare earth minerals, energy consumption, low-wage "clickworkers" creating labels for data sets or doing content moderation are of relevance here (Crawford and Joler 2018;Irani 2016;Veglis 2014;Fang 2019;Casilli 2017). Although "clickwork" is a necessary prerequisite for the application of methods of supervised machine learning, it is associated with numerous social problems (Silberman et al 2018), such as low wages, work conditions and psychological work consequences, which tend to be ignored by the AI community. Finally, yet importantly, not a single guideline raises the issue of public-private partnerships and industry-funded research in the field of AI.…”

“Garbage in, garbage out” revisited: What do machine learning application papers report about human-labeled training data?

“…Blodgett et al conducted a content analysis of how 146 NLP researchers discuss "bias" and found that while this has become a prominent topic in NLP, papers' discussions of motivations and methods around bias "are often vague, inconsistent, and lacking in normative reasoning" (Blodgett et al, 2020, p. 5454). There is also related work in methods papers focused on identifying or preventing "low-effort" responses from crowdworkers (Mozetič et al, 2016;Soberón et al, 2013;Raykar and Yu, 2012), which raise issues around fair labor practices and compensation (Silberman et al, 2018).…”

Crowdsourcing as a Tool for Research: Methodological, Fair, and Political Considerations

“…Practitioners and critics-who are often one and the same-have questioned how representative crowd workers are in comparison to broader populations, and whether insights produced from the data that they provide are generalizable (Williamson, 2016a). Others point to unequal power dynamics between requesters and workers (Samuel, 2018), low compensation rates for completing HITs (Silberman et al, 2018), and the economically precarious circumstances that many crowd workers are in as reasons to question using crowdsourcing in the first place (Berg et al, 2018). And as in other areas of online-mediated research (e.g., Chee et al, 2012), questions of research ethics and the extent to which consent can be validated are not entirely resolved.…”

“…Critics, however, argue that crowd workers' time commitments and legal classification are irrelevant to the issue of fair pay (Silberman et al, 2018). There appears to be a divide between academics whose primary interest in crowdsourcing research is expediency-that is, collecting high-quality data quickly, cheaply, and at scale-and those more concerned with social and economic justice.…”

“…While these pragmatic approaches represent initial steps toward improving conditions for crowdsourced research, there are also a number of proposals that aim for more structural reforms. With respect to compensation rates, some critics have suggested that academics could commit to paying research participants the minimum wage for their location (Silberman et al, 2018). Mandating fair wages-however "fair" may be defined-could lead to more equity for crowd workers, but it still would not resolve the issue of effective wage rates discussed above.…”