Cost-Effective Active Learning from Diverse Labelers

Huang, Sheng-Jun; Chen, Jia-Lve; Mu, Xin; Zhou, Zhi‐Hua

doi:10.24963/ijcai.2017/261

Cited by 56 publications

(58 citation statements)

References 17 publications

(8 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Geva, Saar-Tsechansky, and Lustiger (2019) consider acquiring labels from labelers with varying accuracies and costs based on the estimated effect on generalization error, but do not consider selecting instances for labeling. The most closely related work is by Huang et al (2017): the proposed method estimates workers' labeling accuracies based on a small set of ground-truth data, and then estimates the value of acquiring the label for a given instance from a given worker as the weighted labeling accuracy divide by worker's cost.…”

Section: Related Workmentioning

confidence: 99%

“…Below, we first discuss briefly how we quantify instance usefulness. Because the closest work to the contributions we present here is by Huang et al (2017), we then outline the main elements of the CEAL algorithm (Huang et al 2017) and then describe how our algorithm builds on this contributions.…”

Section: Algorithmmentioning

confidence: 99%

“…In general, any measure for quantifying instance usefulness can be used in our approach. In this paper, our main focus is on a cost-effective selection of labelers, we simply follow the setting in (Huang et al 2017) and use uncertainty sampling as shown in Equation 1. P (y|x j ) is the posterior probability predicted by the classifier trained at current iteration.…”

Section: Instance Usefulnessmentioning

confidence: 99%

See 2 more Smart Citations

Cost-Accuracy Aware Adaptive Labeling for Active Learning

Gao

Saar‐Tsechansky

2020

AAAI

View full text Add to dashboard Cite

Conventional active learning algorithms assume a single labeler that produces noiseless label at a given, fixed cost, and aim to achieve the best generalization performance for given classifier under a budget constraint. However, in many real settings, different labelers have different labeling costs and can yield different labeling accuracies. Moreover, a given labeler may exhibit different labeling accuracies for different instances. This setting can be referred to as active learning with diverse labelers with varying costs and accuracies, and it arises in many important real settings. It is therefore beneficial to understand how to effectively trade-off between labeling accuracy for different instances, labeling costs, as well as the informativeness of training instances, so as to achieve the best generalization performance at the lowest labeling cost. In this paper, we propose a new algorithm for selecting instances, labelers (and their corresponding costs and labeling accuracies), that employs generalization bound of learning with label noise to select informative instances and labelers so as to achieve higher generalization accuracy at a lower cost. Our proposed algorithm demonstrates state-of-the-art performance on five UCI and a real crowdsourcing dataset.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Algorithmmentioning

confidence: 99%

Section: Instance Usefulnessmentioning

confidence: 99%

See 1 more Smart Citation

Cost-Accuracy Aware Adaptive Labeling for Active Learning

Gao

Saar‐Tsechansky

2020

AAAI

View full text Add to dashboard Cite

show abstract

“…In traditional single-label imbalanced data processing methods, relevant studies can be divided into three aspects: algorithm-level methods, cost-sensitive learning methods [6][7][8], and data-level methods. Among algorithm-level methods, the classification algorithm is improved to adapt to imbalanced data sets.…”

Section: Related Workmentioning

confidence: 99%

A Decoupling and Bidirectional Resampling Method for Multilabel Classification of Imbalanced Data with Label Concurrence

Zhou

Dong

et al. 2020

Scientific Programming

View full text Add to dashboard Cite

Label imbalance is one of the characteristics of multilabel data, and imbalanced data seriously affects the performance of the classifiers. In multilabel classification, resampling methods are mostly used to deal with imbalanced problems. Existing resampling methods balance the data by either undersampling or oversampling, which causes overfitting and information loss. Resampling has a significant impact on the minority labels. Furthermore, the high concurrency of majority labels and minority labels in many instances also affects the performance of classification. In this study, we proposed a bidirectional resampling method to decouple multilabel datasets. On one hand, the concurrency of labels can be reduced by setting termination conditions for decoupling, and on the other hand, the loss of instance information and overfitting can be alleviated by combining oversampling and undersampling. By measuring the minority labels of the instances, the instances that have less impact on minority labels are selected to resample. The number of resampling is limited to keep the original distribution of the data during the resampling phase. The experiments on seven benchmark multilabel datasets have proved the effectiveness of the algorithm, especially on datasets with high concurrency of majority labels and minority labels.

show abstract

“…Lin, Mausam, and Weld (2016) tackled the problem of re-active learning, a generalization of active learning that explores the tradeoff between decreasing the noise of the training set via relabeling and increasing the size of the noisy training set by labeling new instances, which introduced two re-active learning algorithms: an extension of uncertainty sampling, and a class of impact sampling algorithms. Huang et al (2017) observed that it is likely that labelers with a low overall quality can provide accurate labels on some specific instances. Based on this fact, they proposed an active selection criterion to evaluate the costeffectiveness of instance-labeler pairs, which ensures that the selected instance is helpful for improving the classification model, and meanwhile the selected labeler can provide an accurate label for the instance with a relatively low cost.…”

Section: Active Learningmentioning

confidence: 99%

Machine Learning with Crowdsourcing: A Brief Summary of the Past Research and Future Directions

Sheng

Zhang

2019

AAAI

View full text Add to dashboard Cite

With crowdsourcing systems, labels can be obtained with low cost, which facilitates the creation of training sets for prediction model learning. However, the labels obtained from crowdsourcing are often imperfect, which brings great challenges in model learning. Since 2008, the machine learning community has noticed the great opportunities brought by crowdsourcing and has developed a large number of techniques to deal with inaccuracy, randomness, and uncertainty issues when learning with crowdsourcing. This paper summarizes the technical progress in this field during past eleven years. We focus on two fundamental issues: the data (label) quality and the prediction model quality. For data quality, we summarize ground truth inference methods and some machine learning based methods to further improve data quality. For the prediction model quality, we summarize several learning paradigms developed under the crowdsourcing scenario. Finally, we further discuss several promising future research directions to attract researchers to make contributions in crowdsourcing.

show abstract

Cost-Effective Active Learning from Diverse Labelers

Cited by 56 publications

References 17 publications

Cost-Accuracy Aware Adaptive Labeling for Active Learning

Cost-Accuracy Aware Adaptive Labeling for Active Learning

A Decoupling and Bidirectional Resampling Method for Multilabel Classification of Imbalanced Data with Label Concurrence

Machine Learning with Crowdsourcing: A Brief Summary of the Past Research and Future Directions

Contact Info

Product

Resources

About