Robust aggregation of GWAP tracks for local image annotation

Abstract

The possibility of assigning labels to localized regions in an image enables flexible image retrieval paradigms. However, the process of automatically segmenting and tagging images is notoriously hard, due to the presence of occlusions, noise, challenging illumination conditions, background clutter, etc. For this reason, human computation has recently emerged as a viable alternative when computer vision algorithms fail to provide a satisfactory answer. For example, Games with a purpose (GWAP) represent a powerful crowdsourcing mechanism to collect implicit annotations from human players. In this paper we consider the problem of aggregating the gaming tracks collected by a GWAP we developed to solve challenging instances of image segmentation problems. In particular we consider the existence of malicious players, who might try to fool the rules of the game to achieve higher rewards. The proposed solution can automatically estimate the reliability of human players, thus identifying cheaters. This information is exploited to aggregate the gaming tracks, thus significantly improving the image segmentation result and the quality of local image annotations.

Machine learning from crowds: A systematic review of its applications

“…The goal here is not to solve problems with subjective components but taking advantage of the crowd to quickly process images that would be used in a machine learning process. We have found the following works related to this problem: obtaining regions of interest from an image (Cabezas et al, ; Su et al, ); image object localization (Salek et al, ); labeling crawled data for object detection (Vijayanarasimhan & Grauman, ); using games for segmenting images (Bernaschina et al, ); maya glyph segmentation (Can et al, ); ambiguity in object segmentation (Gurari et al, ); and salient object detection (Quan et al, ).…”

Designing bots in games with a purpose

“…In particular we consider the existence of malicious players, who might try to fool the rules of the game to achieve higher rewards and thus we need a measure Snippet 1: Fields of a JSON Segmentation Object { " quality " : { " description " : "Estimated quality of the segmentation" , " type " : "number" } , " points " : { " description " : " Points used to reconstruct the binary mask for the segmentation" , " type " : " array " , " items " : { "x" : "number" , "y" : "number" } } , " history " : { " description " : " A l l the points that have been traced by the player " , " type " : " array " , " items " : { " size " : "number" , " color " : "hex color format" , " points " : { " type " : " array " , " items " : { "x" : "number" , "y" : "number" } , "time" : " date " } } } to filter out misleading contributions. Segmentations are used to create binary masks that identify which pixels of the image belong to a given tag/cloth pair, as shown in Figure 5; the algorithm described in [6] is used to aggregate the gaming tracks and compute the best estimate of the mask, starting from all the available segmentations, using a filtered version of the segmentation data.…”

“…The challenge of this approach resides in the fact that different segmentation actions could be available for the same image and only the best ones should be used by the bot; the problem of estimating the contributions' quality was solved as one of the steps necessary to aggregate the traces submitted by the players in a previous work [6] where a novel aggregation algorithm known as Weighted Majority Voting is used to associate a quality value to each available segmentation action and automatically estimate the reliability of human players.…”