Tragedy of the Commons - A Critical Study of Data Quality and Validity Issues in Crowd Work-Based Research

Abstract

Academic scholars have leveraged crowd work platforms such as MTurk to conduct research and collect data. Though prior studies have discussed data quality and validity issues in crowd work via surveys and experiments, they kind of neglected to explore the scholars' and particularly the IRB's ethical concerns in these respects. In this study, we interviewed 17 scholars from six disciplines and 15 IRB directors and analysts in the U.S. to fill this research gap. We identified common themes among our respondents but also discovered distinctive and even opposing views regarding the approval rate, rejection, internal and external research validity. Based on the findings, we discussed a potential Tragedy of the Commons regarding the data quality deterioration and the disciplinary differences regarding validity in crowd work-based research. Finally, we advocated that the IRB's ethical concerns in crowd work-based research should be heard and respected further.

The original sin of crowd work for human subjects research

“…If a scholar chooses to reject these crowd workers, her rejection could stir a reputational risk pertaining to an ethical concern in business because it might affect those crowd workers’ eligibility and welfare in the long term. If a scholar chooses not to reject, as aforementioned, she could incur a “tragedy of the commons” for the research community that depends on crowd work for data collection (Xia, 2022). Third, IRBs would struggle between their concerns about undue influence and scholars’ worry about fair payment because money is the dominant motivation for most crowd workers, which can be coercive to their research participation.…”

“…Along with it, some scholars may have adapted their ethical concerns from academia to business in terms of research payment, like some bioethics scholars who advocated paying medical research participants an hourly wage with the benchmark of a market price in the pharmaceutical industry (Grady, 2001; Resnik, 2001; Cleary et al , 2008). In comparison, IRBs may still be skeptical and against using crowd work for academic research because they perceived that such a marketplace could not guarantee voluntary participation or data quality (Xia, 2022). Such a disparity between scholars and IRBs further creates a delicate tension between scholars and IRBs.…”

“…Such a disparity between scholars and IRBs further creates a delicate tension between scholars and IRBs. For instance, Xia (2022) interviewed 15 scholars and 17 IRB directors and analysts in the USA and found that while the scholars regarded themselves as the protector of crowd workers’ interests, some IRB directors would consider themselves the protector of the scholars’ funding and their institutions’ reputation.…”

“…2. A recent study revealed that some IRB directors regarded it as their responsibility to protect the researchers’ interests and funding in their institution when they conducted crowd work-based research (Xia, 2022). It is because when a researcher rejected some crowd workers (usually with a good reason for these crowd workers’ poor data quality), these crowd workers sometimes protested to the IRB office in that researcher’s institution and even threatened to circulate the researcher’s and the institution’s identity in forums such as Turk Nation and Turkopticon.…”

“…Such a dearth of inquiry into ethical issues in crowd work-based research does not mean their absence or insignificance. For instance, an exploratory study has revealed that many scholars may have overlooked or overestimated crowd work-based research’s validity and data quality (Xia, 2022). The scholars’ unconscious and collective action to avoid rejecting crowd workers may undermine the reputational system to deter data spamming and fraud in crowd work (Xia, 2022).…”