Design Patterns for Hybrid Algorithmic-Crowdsourcing Workflows

Lofi, Christoph; Maarry, Kinda El

doi:10.1109/cbi.2014.16

Cited by 15 publications

(12 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In view of this fact, machine agents must be adaptive to dynamic domains and self-organized crowds. Recent studies have shown that designing workflows that leverage hybrid intelligence for complex tasks are difficult to achieve due to the fact that the current mechanisms are usually deployed in an ad-hoc manner and tailored for specific situations or use cases [51]. In light of these findings, remarkable progress has been made towards more flexible workflows.…”

Section: Integrating Human Inputs Into Ai Systemsmentioning

confidence: 99%

Theoretical Underpinnings and Practical Challenges of Crowdsourcing as a Mechanism for Academic Study

Correia¹,

Jameel²,

Schneider³

et al. 2020

Proceedings of the Annual Hawaii International Conference on System Sciences

View full text Add to dashboard Cite

Researchers in a variety of fields are increasingly adopting crowdsourcing as a reliable instrument for performing tasks that are either complex for humans and computer algorithms. As a result, new forms of collective intelligence have emerged from the study of massive crowd-machine interactions in scientific work settings as a field for which there is no known theory or model able to explain how it really works. Such type of crowd work uses an open participation model that keeps the scientific activity (including datasets, methods, guidelines, and analysis results) widely available and mostly independent from institutions, which distinguishes crowd science from other crowdassisted types of participation. In this paper, we build on the practical challenges of crowd-AI supported research and propose a conceptual framework for addressing the socio-technical aspects of crowd science from a CSCW viewpoint. Our study reinforces a manifested lack of systematic and empirical research of the symbiotic relation of AI with human computation and crowd computing in scientific endeavors.

show abstract

Section: Integrating Human Inputs Into Ai Systemsmentioning

confidence: 99%

Theoretical Underpinnings and Practical Challenges of Crowdsourcing as a Mechanism for Academic Study

Correia¹,

Jameel²,

Schneider³

et al. 2020

Proceedings of the Annual Hawaii International Conference on System Sciences

View full text Add to dashboard Cite

show abstract

“…We note that there are some existing studies [5,9,12,14] that devise machinecrowdsourcing hybrid approaches in other applications, such as, web table matching and information extraction (more details can be referred to Section 7). However, we are the first to devise the hybrid framework in entity type completion.…”

Section: Framework Of Our Crowdsourcing-based Approachmentioning

confidence: 99%

Using Crowdsourcing for Fine-Grained Entity Type Completion in Knowledge Bases

Dong

Fan

et al. 2018

Web and Big Data

View full text Add to dashboard Cite

Recent years have witnessed the proliferation of large-scale Knowledge Bases (KBs). However, many entities in KBs have incomplete type information, and some are totally untyped. Even worse, fine-grained types (e.g., BasketballPlayer) containing rich semantic meanings are more likely to be incomplete, as they are more difficult to be obtained. Existing machine-based algorithms use predicates (e.g., birthPlace) of entities to infer their missing types, and they have limitations that the predicates may be insufficient to infer fine-grained types. In this paper, we utilize crowdsourcing to solve the problem, and address the challenge of controlling crowdsourcing cost. To this end, we propose a hybrid machine-crowdsourcing approach for fine-grained entity type completion. It firstly determines the types of some "representative" entities via crowdsourcing and then infers the types for remaining entities based on the crowdsourcing results. To support this approach, we first propose an embedding-based influence for type inference which considers not only the distance between entity embeddings but also the distances between entity and type embeddings. Second, we propose a new difficulty model for entity selection which can better capture the uncertainty of the machine algorithm when identifying the entity types. We demonstrate the effectiveness of our approach through experiments on real crowdsourcing platforms. The results show that our method outperforms the state-ofthe-art algorithms by improving the effectiveness of fine-grained type completion at affordable crowdsourcing cost.

show abstract

“…This process is based in a 3-layered architecture that defines the set of operations performed by micro-tasks on top of domain ontologies. Lofi et al [22]extensively investigated hybrid crowdsourcing human computation workflows and abstracted generic design patterns. Each design pattern is described and discussed with a special focus on its requirements, constraints, and effects on the overall workflow.…”

Section: Crowdsourcing Task and Workflowmentioning

confidence: 99%

“…For example, Kamar [15] studied how to fuse human and machine contributions to predict the behaviors of workers and presented a principled approach for consensus crowdsourcing. Lofi et al [22] extensively investigated hybrid crowdsourcing human computation workflows and abstracted five generic design patterns, such as Magic Filter, Crowd Trainer, Machine Inprovement, Virtual Worker and High Confidence Switching. Each pattern is described and discussed with a special focus on its requirements, constraints, and effects on the overall workflow and can be extended and combined to support more complex workflows.…”

Section: Crowdsourcing As a Tool For Knowledge Acquisitionmentioning

confidence: 99%

Using hybrid algorithmic-crowdsourcing methods for academic knowledge acquisition

Dong

Ling

et al. 2017

Cluster Comput

View full text Add to dashboard Cite

Scientific literature contains a lot of meaningful objects such as Figures, Tables, Definitions, Algorithms, etc., which are called Knowledge Cells hereafter. An advanced academic search engine which could take advantage of Knowledge Cells and their various relationships to obtain more accurate search results is expected. Further, it's expected to provide a fine-grained search regarding to Knowledge Cells for deep-level information discovery and exploration. Therefore, it is important to identify and extract the Knowledge Cells and their various relationships which are often intrinsic and implicit in articles. With the exponential growth of scientific publications, discovery and acquisition of such useful academic knowledge impose some practical challenges For example, existing algorithmic methods can hardly extend to handle diverse layouts of journals, nor to scale up to process massive documents. As crowdsourcing has become a powerful paradigm for large scale problem-solving especially for tasks that are difficult for computers but easy for human, we consider the prob

show abstract

Design Patterns for Hybrid Algorithmic-Crowdsourcing Workflows

Cited by 15 publications

References 20 publications

Theoretical Underpinnings and Practical Challenges of Crowdsourcing as a Mechanism for Academic Study

Theoretical Underpinnings and Practical Challenges of Crowdsourcing as a Mechanism for Academic Study

Using Crowdsourcing for Fine-Grained Entity Type Completion in Knowledge Bases

Using hybrid algorithmic-crowdsourcing methods for academic knowledge acquisition

Contact Info

Product

Resources

About