OpinionRank: Extracting Ground Truth Labels from Unreliable Expert Opinions with Graph-Based Spectral Ranking

Abstract

As larger and more comprehensive datasets become standard in contemporary machine learning, it becomes increasingly more difficult to obtain reliable, trustworthy label information with which to train sophisticated models. To address this problem, crowdsourcing has emerged as a popular, inexpensive, and efficient data mining solution for performing distributed label collection. However, crowdsourced annotations are inherently untrustworthy, as the labels are provided by anonymous volunteers who may have varying, unreliable expertise. Worse yet, some participants on commonly used platforms such as Amazon Mechanical Turk may be adversarial, and provide intentionally incorrect label information without the end user's knowledge. We discuss three conventional models of the label generation process, describing their parameterizations and the model-based approaches used to solve them. We then propose OpinionRank, a model-free, interpretable, graph-based spectral algorithm for integrating crowdsourced annotations into reliable labels for performing supervised or semi-supervised learning. Our experiments show that OpinionRank performs favorably when compared against more highly parameterized algorithms. We also show that OpinionRank is scalable to very large datasets and numbers of label sources, and requires considerably less computational resources than previous approaches.

Rethinking Noisy Label Models: Labeler-Dependent Noise with Adversarial Awareness

“…For the MNIST dataset, we use auxiliary deep generative models as our SSL algorithm due to its small parameter footprint [50]; for the SVHN dataset, we chose the FixMatch algorithm as representative of the current state-of-the-art for semi-supervised learning [75]. For both datasets, we use OpinionRank as our learning from crowds algorithm due to its nonparametric nature and fast performance [18], and selected DivideMix for learning from noisy labels due to its state-of-the-art performance on a wide variety of noisy labels tasks [44].…”

A Mixed Precision, Multi-GPU Design for Large-scale Top-K Sparse Eigenproblems

“…For many of these applications, hardware accelerators such as Graphics Processing Units (GPUs) and Field Programmable Gate Arrays (FPGAs) are a highly-effective solution, especially when mixed-precision and reduced-precision arithmetic come into play [1]- [6]. As spectral methods become ubiquitous in the large scale graph pipelines of Spectral Clustering [7], Information Retrieval (IR) [8] and ranking [9], such techniques require algorithms that can compute only a subset of the most relevant eigenvalues (i.e. the largest in modulo) and their associated eigenvectors while taking advantage of the sparsity of real-world graphs.…”