Recognizing jumbled images: The role of local and global information in image classification

Abstract

The performance of current state-of-the-art computer vision algorithms at image classification falls significantly short as compared to human abilities. To reduce this gap, it is important for the community to know what problems to solve, and not just how to solve them. Towards this goal, via the use of jumbled images, we strip apart two widely investigated aspects: local and global information in images, and identify the performance bottleneck.Interestingly, humans have been shown to reliably recognize jumbled images. The goal of our paper is to determine a functional model that mimics how humans recognize jumbled images i.e. exploit local information alone, and further evaluate if existing implementations of this computational model suffice to match human performance. Surprisingly, in our series of human studies and machine experiments, we find that a simple bag-of-words based majorityvote-like strategy is an accurate functional model of how humans recognize jumbled images. Moreover, a straightforward machine implementation of this model achieves accuracies similar to human subjects at classifying jumbled images. This indicates that perhaps existing machine vision techniques already leverage local information from images effectively, and future research efforts should be focused on more advanced modeling of global information.

Object Matching Using Boundary Descriptors

“…This suggests that one of the weaknesses of both the BoB and BoN representations is their lack of explicit encoding of the geometric relationship between different descriptor words. Similar findings have been reported in the context of local descriptor based representations of textured objects [9]. We also investigated the possibility of a simple decision level combination of the two representations.…”

Human vs. Computer in Scene and Object Recognition

“…1, 4 th row [3]. Indeed, Parikh [23] showed a a majority-vote accumulation over human classification of the individual blocks is a good predictor of human responses of the entire jumbled images. This dataset contains human performances on 3 image sets: 1) OSR, 384 outdoor scenes from the 8 categories of 8-CAT, 2) ISR, 300 indoor scenes [5] from bathroom, bedroom, dining room, gym, kitchen, living room, theater and staircase categories, and 3) CAL: Caltech objects (50 images from each of 6 categories aeroplane, car-rear, face, ketch, motorbike, and watch).…”

“…Human performance on jumbled images depends on the level of image blocking [23] (here 65%). Model accuracies (trained and tested on jumbled images) are shown in Fig.…”

Improving Image Classification Quality Via Dissimilarity Measure In Non-Euclidean Spaces

“…In the past years, many techniques have been proposed to address the problem of image classification [1][2][3][4][5][6][7][8] . There are two key assumptions in these algorithmic techniques: the first assumption is that images in the database are usually distributed in the Euclidean space, and the second one is that the dissimilarity-based matching is based on the pairwise measure.…”