Hybrid entity clustering using crowds and data

Lee, Jongwuk; Cho, Hyunsouk; Park, Jin-Woo; Cha, Young-Rok; Hwang, Seung-won; Nie, Zaiqing; Wen, Ji-Rong

doi:10.1007/s00778-013-0328-8

Cited by 13 publications

(8 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other research for crowdsourcing ground truth includes entity clustering and disambiguation [18], Twitter entity extraction [13], multilingual entity extraction and paraphrasing [10], and taxonomy creation [11]. However, all of these approaches rely on the assumption that one blackand-white gold standard must exist for every task.…”

Section: Crowdsourcing Ground Truthmentioning

confidence: 99%

Crowdsourcing Ground Truth for Medical Relation Extraction

Dumitrache

Aroyo

Welty

2018

ACM Trans. Interact. Intell. Syst.

View full text Add to dashboard Cite

Cognitive computing systems require human labeled data for evaluation and often for training. The standard practice used in gathering this data minimizes disagreement between annotators, and we have found this results in data that fails to account for the ambiguity inherent in language. We have proposed the CrowdTruth method for collecting ground truth through crowdsourcing, which reconsiders the role of people in machine learning based on the observation that disagreement between annotators provides a useful signal for phenomena such as ambiguity in the text. We report on using this method to build an annotated data set for medical relation extraction for the cause and treat relations, and how this data performed in a supervised training experiment. We demonstrate that by modeling ambiguity, labeled data gathered from crowd workers can (1) reach the level of quality of domain experts for this task while reducing the cost, and (2) provide better training data at scale than distant supervision. We further propose and validate new weighted measures for precision, recall, and F-measure, which account for ambiguity in both human and machine performance on this task.

show abstract

Section: Crowdsourcing Ground Truthmentioning

confidence: 99%

Crowdsourcing Ground Truth for Medical Relation Extraction

Dumitrache

Aroyo

Welty

2018

ACM Trans. Interact. Intell. Syst.

View full text Add to dashboard Cite

show abstract

“…Second, some problems require huge tasks that it is even hard for experts to solve, such as protein folding problem [20]. A large set of answers from nonexperts are aggregated and presented as solution relying on the wisdom of the crowd [21]. Our work successfully unleashed the power of crowds in addressing the license violation problem that has been solely determined through manual inspection by experts.…”

Section: Related Workmentioning

confidence: 99%

Crowdsourcing Identification of License Violations

Lee

Germán

Hwang³

et al. 2015

Journal of Computing Science and Engineering

Self Cite

View full text Add to dashboard Cite

Free and open source software (FOSS) has created a large pool of source codes that can be easily copied to create new applications. However, a copy should preserve copyright notice and license of the original file unless the license explicitly permits such a change. Through software evolution, it is challenging to keep original licenses or choose proper licenses. As a result, there are many potential license violations. Despite the fact that violations can have high impact on protecting copyright, identification of violations is highly complex. It relies on manual inspections by experts. However, such inspection cannot be scaled up with open source software released daily worldwide. To make this process scalable, we propose the following two methods: use machine-based algorithms to narrow down the potential violations; and guide non-experts to manually inspect violations. Using the first method, we found 219 projects (76.6%) with potential violations. Using the second method, we show that the accuracy of crowds is comparable to that of experts. Our techniques might help developers identify potential violations, understand the causes, and resolve these violations.

show abstract

“…Crowdsourcing has grown into a viable alternative to expert ground truth collection, as crowdsourcing tends to be both cheaper and more readily available than domain experts. Experiments have been carried out in a variety of tasks and domains: medical entity extraction [22,53,60], medical relation extraction [29,53], open-domain relation extraction [32], clustering and disambiguation [34], ontology evaluation [42], web resource classification [14] and taxonomy creation [11]. [51] have shown that aggregating the answers of an increasing number of unskilled crowd workers with majority vote can lead to high quality NLP training data.…”

Section: Crowdsourcing Ground Truthmentioning

confidence: 99%

Empirical methodology for crowdsourcing ground truth

Dumitrache

Inel

Timmermans

et al. 2021

View full text Add to dashboard Cite

The process of gathering ground truth data through human annotation is a major bottleneck in the use of information extraction methods for populating the Semantic Web. Crowdsourcing-based approaches are gaining popularity in the attempt to solve the issues related to volume of data and lack of annotators. Typically these practices use inter-annotator agreement as a measure of quality. However, in many domains, such as event detection, there is ambiguity in the data, as well as a multitude of perspectives of the information examples. We present an empirically derived methodology for efficiently gathering of ground truth data in a diverse set of use cases covering a variety of domains and annotation tasks. Central to our approach is the use of CrowdTruth metrics that capture inter-annotator disagreement. We show that measuring disagreement is essential for acquiring a high quality ground truth. We achieve this by comparing the quality of the data aggregated with CrowdTruth metrics with majority vote, over a set of diverse crowdsourcing tasks: Medical Relation Extraction, Twitter Event Identification, News Event Extraction and Sound Interpretation. We also show that an increased number of crowd workers leads to growth and stabilization in the quality of annotations, going against the usual practice of employing a small number of annotators.

show abstract

Hybrid entity clustering using crowds and data

Cited by 13 publications

References 41 publications

Crowdsourcing Ground Truth for Medical Relation Extraction

Crowdsourcing Ground Truth for Medical Relation Extraction

Crowdsourcing Identification of License Violations

Empirical methodology for crowdsourcing ground truth

Contact Info

Product

Resources

About