Human Computation vs. Machine Learning: an Experimental Comparison for Image Classification

Abstract

Image classification is a classical task heavily studied in computer vision and widely required in many concrete scientific and industrial scenarios. Is it better to rely on human eyes, thus asking people to classify pictures, or to train a machine learning system to automatically solve the task? The answer largely depends on the specific case and the required accuracy: humans may be more reliable - especially if they are domain experts - but automatic processing can be cheaper, even if less capable to demonstrate an "intelligent" behaviour.In this paper, we present an experimental comparison of different Human Computation and Machine Learning approaches to solve the same image classification task on a set of pictures used in light pollution research. We illustrate the adopted methods and the obtained results and we compare and contrast them in order to come up with a long term combined strategy to address the specific issue at scale: while it is hard to ensure a long-term engagement of users to exclusively rely on the Human Computation approach, the human classification is indispensable to overcome the "cold start" problem of automated data modelling.

The Patterns of Games for Leading Social Change

“…Unlike the science fiction doom portrayed future, where humanity fights the growing threat of a sentient artificial intelligence or a more powerful computer, it is more likely that there will be a harmony between creative problem‐solving employing humanity's unique characteristics and the propensities of computing. This is hinted at by findings that machine learning remains dependent on human input (Calegari et al., 2018).…”