Large-scale linked data integration using probabilistic reasoning and crowdsourcing

Abstract

We tackle the problems of semiautomatically matching linked data sets and of linking large collections of Web pages to linked data. Our system, ZenCrowd, (1) uses a three-stage blocking technique in order to obtain the best possible instance matches while minimizing both computational complexity and latency, and (2) identifies entities from natural language text using state-of-the-art techniques and automatically connects them to the linked open data cloud. First, we use structured inverted indices to quickly find potential candidate results from entities that have been indexed in our system. Our system then analyzes the candidate matches and refines them whenever deemed necessary using computationally more expensive queries on a graph database. Finally, we resort to human computation by dynamically generating crowdsourcing tasks in case the algorithmic components fail to come up with convincing results. We integrate all results from the inverted indices, from the graph database and from the crowd using a probabilistic framework in order to make sensible decisions about candidate matches and to identify unreliable human workers. In the following, we give an overview of the architecture of our system and describe in detail our novel three-stage blocking technique and our probabilistic decision framework. We also report on a series of experimental results on a standard data set, showing that our system can achieve a 95 % average accuracy on

Using microtasks to crowdsource DBpedia entity classification: A study in workflow design

“…The other approach is to use a machine to narrow down the possible options and then employ the crowd to validate or chose the best matching one. As an example, the work presented in [23] employs a machine-based algorithm to classify entities along with calculating a confidence score. The authors suggest that the label crowdsourcing is required only for entities with low confidence scores produced by the classifier.…”

“…We used this level for all three workflows. Aggregation: For the T2 tasks we used the default option (aggregation='agg') 23 , as the task is to choose from a set of pre-defined options. For T1, we looked at the first three answers (aggregation='agg_3') based on 11 judgments.…”

Pay-as-you-go Configuration of Entity Resolution

“…ZenCrowd [9] identifies pairs of instances in linked data using two levels of blocking to identify candidate pairs for confirmation by the crowd. A probabilistic factor graph accumulates evidence from different sources, from which a probability is derived that a candidate pair is correct.…”

Improving Paid Microtasks through Gamification and Adaptive Furtherance Incentives

“…Many projects have already demonstrated substantial success in applying this idea to crowdsourcing settings; this applies most prominently for games-with-a purpose (GWAPs) [27], which build a game narrative around human computation tasks such as image labeling [26], protein folding, 5 or language translation. 6 Similarly to the concerns raised in the context of external rewards and incentivisation [18], gamification has been seen, in some context, to undermine intrinsic benefits by subjugating and trivialising contributions into simple game goals and achievements. 7 This effect has been called overjustification and has been the subject of various studies with intriguing results; while some negative effects of overjustification have been recurrently reproduced, current research acknowledges the fact that its prevalence seems to be highly dependent on context and that, in most cases, extrinsic rewards complement rather than hamper intrinsic motivations for participating [5,22].…”