Human Computation vs. Machine Learning: an Experimental Comparison for Image Classification

Abstract

Image classification is a classical task heavily studied in computer vision and widely required in many concrete scientific and industrial scenarios. Is it better to rely on human eyes, thus asking people to classify pictures, or to train a machine learning system to automatically solve the task? The answer largely depends on the specific case and the required accuracy: humans may be more reliable - especially if they are domain experts - but automatic processing can be cheaper, even if less capable to demonstrate an "intelligent" behaviour.In this paper, we present an experimental comparison of different Human Computation and Machine Learning approaches to solve the same image classification task on a set of pictures used in light pollution research. We illustrate the adopted methods and the obtained results and we compare and contrast them in order to come up with a long term combined strategy to address the specific issue at scale: while it is hard to ensure a long-term engagement of users to exclusively rely on the Human Computation approach, the human classification is indispensable to overcome the "cold start" problem of automated data modelling.

Remote sensing of night lights: A review and an outlook for the future

“…2018). Several groups have used the "Cities at Night" as training sample for computer vision proposes, including Minh Hieu (2016), Calegari et al (2018) and Sadler (2018). Based on this catalogue it can be seen that ISS night-time photos are not representing all parts of the world, and are more common in the urban areas of North America, Europe, the Middle East, eastern China and Japan (Figure 15a).…”

An Incremental Truth Inference Approach to Aggregate Crowdsourcing Contributions in Games with a Purpose

“…Therefore, whenever an assessment of the task difficulty is required, the number of collected contributions can be adopted as a proxy measure. In our previous work [18] we indeed demonstrated that this empirical measure of difficulty is highly correlated with the (lack of) confidence value resulting from machine learning classifiers applied to the same data.…”

“…To evaluate the proposed truth inference algorithm we performed a comparative assessment with alternative solutions, on the basis of the data collected through two different GWAPs: the LCV Game [19] and Night Knights [18].…”

Who is this Explanation for? Human Intelligence and Knowledge Graphs for eXplainable AI

“…Indeed, it can happen that what is "difficult" to predict for an algorithm (i.e. predictions with low confidence metrics) is also difficult for humans to judge; the case of questionable image classification is illustrated in [26], where the correspondence exists between low-confidence machine classifications and user disagreement. The correlation between human and machine predictions and their respective confidence/reliability can be exploited to understand the reasons behind a model and can therefore improve both the modelling phase (by incorporating additional human knowledge in training) and the generation of explanations (which can be closer to human understanding).…”