Auto-experimentation of KDD Workflows Based on Ontological Planning

Serban, Floarea

doi:10.1007/978-3-642-17749-1_22

Cited by 4 publications

(2 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The PPLib Programming Library is modeled after the generic human communication patterns described in speech acts [Searle 1969;Winograd and Flores 1986]. Comparable to Turkit , one develops code that asks crowd workers questions and acts upon their answers.…”

Section: The Pplib Programming Librarymentioning

confidence: 99%

PPLib

Boer

Bernstein

2016

ACM Trans. Intell. Syst. Technol.

View full text Add to dashboard Cite

Crowdsourcing is increasingly being adopted to solve simple tasks such as image labeling and object tagging, as well as more complex tasks, where crowd workers collaborate in processes with interdependent steps. For the whole range of complexity, research has yielded numerous patterns for coordinating crowd workers in order to optimize crowd accuracy, efficiency, and cost. Process designers, however, often don't know which pattern to apply to a problem at hand when designing new applications for crowdsourcing.In this article, we propose to solve this problem by systematically exploring the design space of complex crowdsourced tasks via automated recombination and auto-experimentation for an issue at hand. Specifically, we propose an approach to finding the optimal process for a given problem by defining the deep structure of the problem in terms of its abstract operators, generating all possible alternatives via the (re)combination of the abstract deep structure with concrete implementations from a Process Repository, and then establishing the best alternative via auto-experimentation.To evaluate our approach, we implemented PPLib (pronounced "People Lib"), a program library that allows for the automated recombination of known processes stored in an easily extensible Process Repository. We evaluated our work by generating and running a plethora of process candidates in two scenarios on Amazon's Mechanical Turk followed by a meta-evaluation, where we looked at the differences between the two evaluations. Our first scenario addressed the problem of text translation, where our automatic recombination produced multiple processes whose performance almost matched the benchmark established by an expert translation. In our second evaluation, we focused on text shortening; we automatically generated 41 crowd process candidates, among them variations of the well-established Find-Fix-Verify process. While Find-Fix-Verify performed well in this setting, our recombination engine produced five processes that repeatedly yielded better results. We close the article by comparing the two settings where the Recombinator was used, and empirically show that the individual processes performed differently in the two settings, which led us to contend that there is no unifying formula, hence emphasizing the necessity for recombination. ACM Reference Format:Patrick M. de Boer and Abraham Bernstein. 2016. PPLib: Toward the automated generation of crowd computing programs using process recombination and auto-experimentation.

show abstract

Section: The Pplib Programming Librarymentioning

confidence: 99%

PPLib

Boer

Bernstein

2016

ACM Trans. Intell. Syst. Technol.

View full text Add to dashboard Cite

show abstract

“…Auto-Experimentation Engines, systems that autonomously plan and run the evaluation of a given set of hypothesis', may be adopted towards this cause. They have been used successfully in biology [19], machine learning [31], and, most recently, crowd computing [7]. Specifically, de Boer and Bernstein proposed PPLib, a method and tool capable of automating large parts of crowd process design by viewing crowd process suitability to a given problem as a hypothesis to be answered through Auto-Experimentation.…”

Section: Auto-experimentation For Crowd Computingmentioning

confidence: 99%

Efficiently Identifying a Well-Performing Crowd Process for a Given Problem

Boer

Bernstein

2017

Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing

View full text Add to dashboard Cite

With the increasing popularity of crowdsourcing and crowd computing, the question of how to select a well-performing crowd process for a problem at hand is growing ever more important. Prior work casted crowd process selection to an optimization problem, whose solution is the crowd process performing best for a user's problem. However, existing approaches require users to probabilistically model aspects of the problem, which may entail a substantial investment of time and may be error-prone. We propose to use black-box optimization instead, a family of techniques that do not require probabilistic modelling by the end user. Specifically, we adopt Bayesian Optimization to approximate the maximum of a utility function quantifying the user's (business-) objectives while minimizing search cost. Our approach is validated in a simulation and three real-world experiments. The black-box nature of our approach may enable us to reduce the entry barrier for efficiently building crowdsourcing solutions.

show abstract

Performance Measurement of Building Sheet-Metal Ductwork Prefabrication under Batch Production Settings

Said

Kandimalla

2018

J. Constr. Eng. Manage.

View full text Add to dashboard Cite

Auto-experimentation of KDD Workflows Based on Ontological Planning

Cited by 4 publications

References 10 publications

PPLib

PPLib

Efficiently Identifying a Well-Performing Crowd Process for a Given Problem

Performance Measurement of Building Sheet-Metal Ductwork Prefabrication under Batch Production Settings

Contact Info

Product

Resources

About