Comparing machines and humans on a visual categorization test

Abstract

Automated scene interpretation has benefited from advances in machine learning, and restricted tasks, such as face detection, have been solved with sufficient accuracy for restricted settings. However, the performance of machines in providing rich semantic descriptions of natural scenes from digital images remains highly limited and hugely inferior to that of humans. Here we quantify this "semantic gap" in a particular setting: We compare the efficiency of human and machine learning in assigning an image to one of two categories determined by the spatial arrangement of constituent parts. The images are not real, but the category-defining rules reflect the compositional structure of real images and the type of "reasoning" that appears to be necessary for semantic parsing. Experiments demonstrate that human subjects grasp the separating principles from a handful of examples, whereas the error rates of computer programs fluctuate wildly and remain far behind that of humans even after exposure to thousands of examples. These observations lend support to current trends in computer vision such as integrating machine learning with parts-based modeling.abstract reasoning | human learning | pattern recognition I mage interpretation, effortless and instantaneous for people, remains a fundamental challenge for artificial intelligence. The goal is to build a "description machine" that automatically annotates a scene from image data, detecting and describing objects, relationships, and context. It is generally acknowledged that building such a machine is not possible with current methodology, at least when measuring success against human performance.Some well-circumscribed problems have been solved with sufficient speed and accuracy for real-world applications. Almost every digital camera on the market today carries a face detection algorithm that allows one to adjust the focus according to the presence of humans in the scene; and machine vision systems routinely recognize flaws in manufacturing, handwritten characters, and other visual patterns in controlled industrial settings.However, such cases usually involve a single quasi-rigid object or an arrangement of a few discernible parts and thus do not display many of the complications of full-scale "scene understanding." Moreover, achieving high accuracy usually requires intense "training" with gigantic amounts of data. Systems that attempt to deal with multiple object categories, high intraclass variability, occlusion, context, and unanticipated arrangements, all of which are easily handled by people, typically perform poorly. Such visual complexity seems to require a form of global reasoning that uncovers patterns and generates high-level hypotheses from local measurements and prior world knowledge.In order to go beyond general observation and speculation, we have designed a controlled experiment to measure the difference in performance between computer programs and human subjects. The Synthetic Visual Reasoning Test (SVRT) is a series of 23 classification problems involvi...

25 Years of CNNs: Can We Compare to Human Abstraction Capabilities?

“…LeNet has an average accuracy of 0.95 and GoogLeNet achieves practically perfect accuracy. Both also compare very favorably to the method presented by Fleuret et al [3] which achieves a mean accuracy of 0.86 on this subset of problems. We will discuss those two subsets of problems in the following sections in more detail.…”

“…At first glance, CNNs do not seem to have made much progress over the last 25 years with the types of problems we tested, and even compare very unfavorably to the boosting method presented by Fleuret et al [3]. The average accuracy of GoogLeNet even decreased slightly compared to LeNet.…”

“…The reported accuracy of the network is the proportion of correctly classified images to the number of all tested images. For three problems (3, 11, 13) from Fleuret et al [3] we could not generate images of the correct size. Since we do not think it will influence the overall conclusion, we excluded those problems from our evaluation.…”

“…Since we do not think it will influence the overall conclusion, we excluded those problems from our evaluation. Table 1 gives an overview of the achieved accuracy of both tested network architectures, the method presented by Fleuret et al [3], and human test subjects. In addition, the table gives a short description of the properties which are used to differentiate the two classes.…”

“…We use the framework presented by Fleuret et al [3] to generate our dataset consisting of 23 different problems which are briefly summarized as follows: Each problem consists of two classes of images. Images of the first class exhibit some abstract property which is not present in images of the second class and vice versa.…”