Clickstream Analysis for Crowd-Based Object Segmentation with Confidence

Abstract

Abstract-With the rapidly increasing interest in machine learning based solutions for automatic image annotation, the availability of reference annotations for algorithm training is one of the major bottlenecks in the field. Crowdsourcing has evolved as a valuable option for low-cost and large-scale data annotation; however, quality control remains a major issue which needs to be addressed. To our knowledge, we are the first to analyze the annotation process to improve crowd-sourced image segmentation. Our method involves training a regressor to estimate the quality of a segmentation from the annotator's clickstream data. The quality estimation can be used to identify spam and weight individual annotations by their (estimated) quality when merging multiple segmentations of one image. Using a total of 29,000 crowd annotations performed on publicly available data of different object classes, we show that (1) our method is highly accurate in estimating the segmentation quality based on clickstream data, (2) outperforms state-of-the-art methods for merging multiple annotations. As the regressor does not need to be trained on the object class that it is applied to it can be regarded as a low-cost option for quality control and confidence analysis in the context of crowd-based image annotation.

Exploiting the potential of unlabeled endoscopic video data with self-supervised learning

“…-DS COCO L: All 2,818 images of cats from the COCO dataset [14] and the corresponding segmentations. We chose cats as target class because unlike other classes of the COCO dataset, the corresponding images do not suffer from poor references or ambiguities [10] and have the target object in the foreground (similar to medical instruments in endoscopic data). Note, however, that the color distribution of DS COCO L is comparable to that of the whole COCO dataset.…”

A crowdsourcing semi-automatic image segmentation platform for cell biology

“…[30]. [40] is one of the few studies that correlated the mouse dynamics and clicks stream data with annotation quality in crowdsourced image segmentation. In that study, a regression model was trained to estimate the quality of annotations with respect to the features extracted from the clicks stream, i.e.…”

Toward a standard ontology of surgical process models

“…The model has been applied to several hundreds liver tumor patients, and is currently being extended to applications in renal surgery, including intraoperative process models based on LapOntoSPM. In this context, new methods for large-scale medical data annotation based on crowdsourcing have been developed [25,40,41]. An implementation of the system is publicly available [69].…”