Laconic Image Classification: Human vs. Machine Performance

Abstract

We propose laconic classification as a novel way to understand and compare the performance of diverse image classifiers. The goal in this setting is to minimise the amount of information (aka. entropy) required in individual test images to maintain correct classification. Given a classifier and a test image, we compute an approximate minimal-entropy positive image for which the classifier provides a correct classification, becoming incorrect upon any further reduction. The notion of entropy offers a unifying metric that allows to combine and compare the effects of various types of reductions (e.g., crop, colour reduction, resolution reduction) on classification performance, in turn generalising similar methods explored in previous works. Proposing two complementary frameworks for computing the minimal-entropy positive images of both human and machine classifiers, in experiments over the ILSVRC test-set, we find that machine classifiers are more sensitive entropy-wise to reduced resolution (versus cropping or reduced colour for machines, as well as reduced resolution for humans), supporting recent results suggesting a texture bias in the ILSVRC-trained models used. We also find, in the evaluated setting, that humans classify the minimalentropy positive images of machine models with higher precision than machines classify those of humans. CCS CONCEPTS • Information systems → Multimedia information systems; • Computing methodologies → Neural networks.

Comparison of Visual and Visual–Tactile Inspection of Aircraft Engine Blades

“…While classification problems are an integral part of machine learning [27], there has been little emphasis on measuring and comparing the classification accuracies of humans with those of software or other technological systems. Previous studies compared the performance of humans and software in text classification problems [28,29] and image recognition and classification tasks [30,31], assessing effects such as image quality and visual distortion. However, the accuracy of defect classification has not yet been analysed.…”