Making Hard Fun in Crowdsourced Model Checking -- Balancing Crowd Engagement and Efficiency to Maximize Output in Proof by Games

Moffitt, Kerry; Ostwald, John; Watro, Ron; Church, E.

doi:10.1109/csi-se.2015.13

Cited by 3 publications

(3 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These range from designing RNA folds [13] to improving cropland coverage maps [24] to building 3D models of neurons [9] to software verification [16]. …”

Section: Introductionmentioning

confidence: 99%

To Three or not to Three

Gaston

Cooper

2017

Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

While many popular casual games use three-star systems, which give players up to three stars based on their performance in a level, this technique has seen limited application in human computation games (HCGs). This gives rise to the question of what impact, if any, a three-star system will have on the behavior of players in HCGs. In this work, we examined the impact of a three-star system implemented in the protein folding HCG Foldit. We compared the basic game’s introductory levels with two versions using a three-star system, where players were rewarded with more stars for completing levels in fewer moves. In one version, players could continue playing levels for as many moves as they liked, and in the other, players were forced to reset the level if they used more moves than required to achieve at least one star on the level. We observed that the three-star system encouraged players to use fewer moves, take more time per move, and replay completed levels more often. We did not observe an impact on retention. This indicates that three-star systems may be useful for re-enforcing concepts introduced by HCG levels, or as a flexible means to encourage desired behaviors.

show abstract

“…These range from designing RNA folds [13] to improving cropland coverage maps [24] to building 3D models of neurons [9] to software verification [16]. …”

Section: Introductionmentioning

confidence: 99%

To Three or not to Three

Gaston

Cooper

2017

Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…As discussed in the introduction, several projects investigated the idea of crowdsourcing verification tasks [7,8,12,13,21,23,30]. The main focus of these projects is to investigate to gamification potential and how to create a problem representation that is appealing to gamers.…”

Section: Related Workmentioning

confidence: 99%

“…DARPA ran a project on crowdsourcing verification problems with several teams developing games that crowdsource different aspects of various verification approaches. For example, one of the developed games, GhostMap [23,30], crowdsources the problem of finding suitable refinements in a CEGAR-based model checker. Another game, Xylem [13], crowdsources predicate discovery to assist the abstract interpretation tool Frama-C. Other tools, like BinaryFission [12] crowdsource invariant discovery to obtain more readable invariants than existing machine learning approaches, and Paradox or Pipejam [8] turn constraint systems from type checking into puzzle games.…”

Section: Introductionmentioning

confidence: 99%

Abduction by Non-Experts

Bjørner¹,

Jovanović²,

Lepoint³

et al.

Kalpa Publications in Computing

View full text Add to dashboard Cite

Crowdsourcing promises to quasi-automate tasks that cannot be automated otherwise. Success stories like natural language translation or recognition of cats in images show that carefully crafted crowdsourcing tasks solve large problem instances which could not be solved otherwise. To utilize crowdsourcing, one has to define the problem in a way that is easy to split into small tasks, that the tasks are easy to solve for humans and hard to solve for a machine, and that the machine can efficiently check if the solution is correct.In this paper we discuss a novel approach of using crowdsourcing to assist software verification. We argue that Horn clauses form a good base for crowdsourcing since they are easy to subdivide, and that logic abduction is a suitable task since it is hard to find abductive inferences for Horn clauses automatically, but it is easy to check if an inference makes a Horn clause valid. We describe a prototype implementation, we show how crowdsourcing integrates in the verification process, and present preliminary results.

show abstract