Assessing Neural Network Scene Classification from Degraded Images

Abstract

Scene recognition is an essential component of both machine and biological vision. Recent advances in computer vision using deep convolutional neural networks (CNNs) have demonstrated impressive sophistication in scene recognition, through training on large datasets of labeled scene images (Zhou et al. 2018, 2014). One criticism of CNN-based approaches is that performance may not generalize well beyond the training image set (Torralba and Efros 2011), and may be hampered by minor image modifications, which in some cases are barely perceptible to the human eye (Goodfellow et al. 2015; Szegedy et al. 2013). While these "adversarial examples" may be unlikely in natural contexts, during many real-world visual tasks scene information can be degraded or limited due to defocus blur, camera motion, sensor noise, or occluding objects. Here, we quantify the impact of several image degradations (some common, and some more exotic) on indoor/outdoor scene classification using CNNs. For comparison, we use human observers as a benchmark, and also evaluate performance against classifiers using limited, manually selected descriptors. While the CNNs outperformed the other classifiers and rivaled human accuracy for intact images, our results show that their classification accuracy is more affected by image degradations than human observers. On a practical level, however, accuracy of the CNNs remained well above chance for a wide range of image manipulations that disrupted both local and global image statistics. We also examine the level of image-by-image agreement with human observers, and find that the CNNs' agreement with observers varied as a function of the nature of image manipulation. In many cases, this agreement was not substantially different from the level one would expect to observe for two independent classifiers. Together, these results suggest that CNN-based scene classification techniques are relatively robust to several image degradations. However, the pattern of classifications obtained for ambiguous images does not appear to closely reflect the strategies employed by human observers.

Do Humans and Deep Convolutional Neural Networks Use Visual Information Similarly for the Categorization of Natural Scenes?

“…Then, we proceeded to compare the relative accuracy between artificial and human performance, and to analyze the level of agreement (or confusion) at the image-level between humans and CNNs. In order to compare accuracy, we followed the approach suggested by Tadros et al (2019). First, human and CNN accuracy data were aggregated per type and level of degradation.…”

“…In order to compare accuracy, we followed the approach suggested by Tadros et al. ( 2019 ). First, human and CNN accuracy data were aggregated per type and level of degradation.…”

“…A value of 1 indicates equal accuracy by humans and CNNs, while a value larger than 1 indicates better accuracy for CNNs, and a value smaller than 1 indicates a better performance for humans. Following Tadros et al (2019), and using the log method described in Katz, Baptista, Azen, and Pike (1978) and Koopman (1984), we calculated standard deviation and p-values, corrected (Bonferroni method) for four multiple comparisons, corresponding to the four degradation levels, which were used for each degradation type. Additionally, we calculated the agreement in the image-level responses between humans and CNNs.…”

“…Recently, some studies have directly compared visual processing in humans and CNNs, focusing on the final performance achieved by the classificator rather than on the internal (brain or layers) visual input representation that is achieved (Dodge & Karam, 2019;Geirhos et al, 2018;Kubilius, Bracci, & Op de Beeck, 2016;Rajalingham et al, 2018;Tadros et al, 2019). When directly comparing humans and CNNs, Dodge and Karam (2019) and Tadros et al (2019) observed that while CNN visual classification of intact stimuli is on a par with that of human participants, it drops significantly with degraded pictures, and that CNN errors have little correlation with human ones. A similar result was observed by Geirhos et al (2018), who additionally observed that, while retraining the CNN on the distortion type it would be tested on ameliorated its performance, this improvement did not generalize to other distortion types.…”

“…When directly comparing humans and CNNs, Dodge and Karam ( 2019 ) and Tadros et al. ( 2019 ) observed that while CNN visual classification of intact stimuli is on a par with that of human participants, it drops significantly with degraded pictures, and that CNN errors have little correlation with human ones. A similar result was observed by Geirhos et al.…”