CGLAD: Using GLAD in Crowdsourced Large Datasets

Rodrigo, Enrique González; Aledo, Juan A.; Gámez, José A.

doi:10.1007/978-3-030-03493-1_81

Cited by 2 publications

(1 citation statement)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other authors highlight that estimating the difficulty of annotators could improve the reliability of the annotator performance estimation, which could lead to more powerful models (Aung & Whitehill, ; Cao et al, ; Duan et al, ; Nguyen‐Dinh et al, ; Ni et al, ; Wan & Aggarwal, ). There are some classic crowdsourcing algorithms that take into account instance difficulty, such as (Donmez, Carbonell, & Schneider, ; Welinder, Branson, Perona, & Belongie, ; Whitehill, fan Wu, Bergsma, Movellan, & Ruvolo, ), which could be used directly or as a basis for the development of new algorithms that consider the restrictions of the problem at hand, such as the scalability to a large number of examples or the time complexity of the algorithm (Rodrigo, Aledo, & Gámez, ).…”

Section: Future Research In the Fieldmentioning

confidence: 99%

Machine learning from crowds: A systematic review of its applications

Rodrigo

Aledo

Gámez

2018

WIREs Data Min & Knowl

Self Cite

View full text Add to dashboard Cite

Crowdsourcing opens the door to solving a wide variety of problems that previously were unfeasible in the field of machine learning, allowing us to obtain relatively low cost labeled data in a small amount of time. However, due to the uncertain quality of labelers, the data to deal with are sometimes unreliable, forcing practitioners to collect information redundantly, which poses new challenges in the field. Despite these difficulties, many applications of machine learning using crowdsourced data have recently been published that achieved state of the art results in relevant problems. We have analyzed these applications following a systematic methodology, classifying them into different fields of study, highlighting several of their characteristics and showing the recent interest in the use of crowdsourcing for machine learning. We also identify several exciting research lines based on the problems that remain unsolved to foster future research in this field.

show abstract