Our experiments show that the detection of errors in automatic cortical surface delineations generated by anonymous untrained workers is feasible. Future work will focus on increasing the sensitivity of our method further, such that the error detection tasks can be handled exclusively by the crowd and expert resources can be focused on error correction.
Abstract-With the rapidly increasing interest in machine learning based solutions for automatic image annotation, the availability of reference annotations for algorithm training is one of the major bottlenecks in the field. Crowdsourcing has evolved as a valuable option for low-cost and large-scale data annotation; however, quality control remains a major issue which needs to be addressed. To our knowledge, we are the first to analyze the annotation process to improve crowd-sourced image segmentation. Our method involves training a regressor to estimate the quality of a segmentation from the annotator's clickstream data. The quality estimation can be used to identify spam and weight individual annotations by their (estimated) quality when merging multiple segmentations of one image. Using a total of 29,000 crowd annotations performed on publicly available data of different object classes, we show that (1) our method is highly accurate in estimating the segmentation quality based on clickstream data, (2) outperforms state-of-the-art methods for merging multiple annotations. As the regressor does not need to be trained on the object class that it is applied to it can be regarded as a low-cost option for quality control and confidence analysis in the context of crowd-based image annotation.
Abstract. In recent global business environments, collaborations among organisations raise an increased demand for swift establishment. Such collaborations are formed between organisations entering Virtual Organizations (VOs), crossing geographic borders and frequently without prior experience of the other partner's previous performance. In VOs, every participant risks engaging with partners who may exhibit unexpected fraudulent or otherwise untrusted behaviour. In order to cope with this risk, the STochastic REputation system (STORE) was designed to provide swift, automated decision support for selecting partner organisations in the early stages of the VO's formation. The contribution of this paper first consists of a multi-agent simulation framework design and implementation to evaluate the STORE reputation system. This framework is able to simulate dynamic agent behaviour, agents hereby representing organisations, and to capture the business context of different VO application scenarios. A configuration of agent classes is a powerful tool to obtain not only well or badly performing agents for simulation scenarios, but also agents which are specialized in particular VO application domains or even malicious agents, attacking the VO community. The second contribution comprises of STORE's evaluation in two simulation scenarios, set in the VO application domains of Collaborative Engineering and Ad-hoc Service provisioning. Besides the ability to clearly distinguish between agents of different classes according to their reputation, the results prove STORE's ability to take an agent's dynamic behaviour into account. The simulation results show, that STORE solves the difficult task of selecting the most trustworthy partner for a particular VO application domain from a set of honest agents that are specialized in a wide spread of VO application domains.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.