Towards Fair Truth Discovery from Biased Crowdsourced Answers

Li, Yanying; Sun, Haipei; Wang, Wendy Hui

doi:10.1145/3394486.3403102

Cited by 29 publications

(25 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fair task assignment and truth discovery (Goel and Faltings, 2019;Li et al, 2020d) are different subproblems in the same area, focused on the subdivision of work and the aggregation of answers in crowdsourcing. Here fairness may be intended concerning errors in the aggregated answer, requiring errors to be balanced across subpopulations of interest, or in terms of the work load imposed to workers.…”

Section: Taskmentioning

confidence: 99%

See 1 more Smart Citation

Algorithmic Fairness Datasets: the Story so Far

Fabris,

Messina,

Silvello

et al. 2022

Preprint

View full text Add to dashboard Cite

Data-driven algorithms are being studied and deployed in diverse domains to support critical decisions, directly impacting on people's wellbeing. As a result, a growing community of algorithmic fairness researchers has been investigating the equity of existing algorithms and proposing novel ones, advancing the understanding of the risks and opportunities of automated decision-making for different populations. Algorithmic fairness progress hinges on data, which can be used appropriately only if adequately documented. Unfortunately, the algorithmic fairness community, as a whole, suffers from a collective data documentation debt caused by a lack of information on specific resources (opacity) and scatteredness of available information (sparsity). In this work, we survey over two hundred datasets employed in algorithmic fairness research, producing standardized and searchable documentation for each of them, along with in-depth documentation for the three most popular fairness datasets, namely Adult, COMPAS and German Credit. These documentation efforts support multiple contributions. Firstly, we summarize the merits and limitations of popular algorithmic fairness datasets, questioning their suitability as general-purpose fairness benchmarks. Secondly, we document hundreds of available alternatives, annotating their domain and supported fairness tasks, to assist dataset users in task-oriented and domain-oriented search. Finally, we analyze these resources from the perspective of five important data curation topics: anonymization, consent, inclusivity, labeling of sensitive attributes, and transparency. We discuss different approaches and levels of attention to these topics, making them tangible, and distill them into a set of best practices for the curation of novel datasets.

show abstract

Section: Taskmentioning

confidence: 99%

“…-Tasks in fairness literature: fair truth discovery (Li et al, 2020d), fair task assignment (Li et al, 2020d;Goel and Faltings, 2019) (for crowdsourced judgements). -Data spec: judge-defendant pair.…”

Section: A40 Credit Card Defaultmentioning

confidence: 99%

Algorithmic Fairness Datasets: the Story so Far

Fabris,

Messina,

Silvello

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Human-based computation has been studied for decades to address the problems that are hard to solve totally algorithmically, such as building hierarchy [5,34], entity resolution [37,38,40,42], object categorization [23,47], data filtering [30,32], data labeling [10], SQL-like query processing [9,18,36], and data cleaning [44]. Problems in human-based computation have attracted considerable attention in the database and data mining areas [2,15,20,22,24,41,48]. Among the above problems, the crowd-based filtering problem [30,32] is closest to our work, which aims to filter objects based on a set of properties with the minimum cost while ensuring accuracy.…”

Section: Related Workmentioning

confidence: 99%

Noisy Interactive Graph Search

Cong

Han

Huang

et al. 2022

Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

The interactive graph search (IGS) problem aims to locate an initially unknown target node leveraging human intelligence. In IGS, we can gradually find the target node by sequentially asking humans some reachability queries like "is the target node reachable from a given node 𝑥?". However, human workers may make mistakes when answering these queries. Motivated by this concern, in this paper, we study a noisy version of the IGS problem. Our objective in this problem is to minimize the query complexity while ensuring accuracy. We propose a method to select the query node such that we can push the search process as much as possible and an online method to infer which node is the target after collecting a new answer. By rigorous theoretical analysis, we show that the query complexity of our approach is near-optimal up to a constant factor. The extensive experiments on two real datasets also demonstrate the superiorities of our approach. CCS CONCEPTS• Information systems → Crowdsourcing; • Theory of computation → Graph algorithms analysis.

show abstract

“…For example, aggregating the medical data from workers helps discover the effects of new drugs but may also disclose their health conditions. Accordingly, a series of privacy-preserving truth discovery approaches have been proposed [13][14][15][16][17][18][19][20][21][22], where secure multiparty computation (MPC) [23][24][25][26] or differential privacy [15,[27][28][29][30][31] are employed to preserve the privacy of the participant workers.…”

Section: Introduction E Proliferation Of Information Techniquesmentioning

confidence: 99%

Achieving Private and Fair Truth Discovery in Crowdsourcing Systems

Wang

Cheng

et al. 2022

Security and Communication Networks

View full text Add to dashboard Cite

Nowadays, crowdsourcing has witnessed increasing popularity as it can be adopted to solve many challenging question-answering tasks. One of the most significant problems in crowdsourcing is truth discovery, which aims to find reliable information from conflict answers provided by different workers. Despite the effectiveness for providing reliable aggregated results, existing works on truth discovery either fall short of preserving the workers’ privacy or fail to consider the unfairness issue in their design. In light of this deficiency, we propose a novel private and fair truth discovery approach called PFTD, which is implemented by two non-colluding cloud servers and leverages the Paillier cryptosystem. This approach not only preserves the privacy of the answers of each worker, but also addresses the unfairness issue in crowdsourcing. Extensive experiments conducted on both real and synthetic datasets demonstrate the effectiveness of our proposed PFTD approach.

show abstract

Towards Fair Truth Discovery from Biased Crowdsourced Answers

Cited by 29 publications

References 19 publications

Algorithmic Fairness Datasets: the Story so Far

Algorithmic Fairness Datasets: the Story so Far

Noisy Interactive Graph Search

Achieving Private and Fair Truth Discovery in Crowdsourcing Systems

Contact Info

Product

Resources

About