Qualitative similarities and differences in visual object representations between brains and deep networks

Abstract

Deep neural networks have revolutionized computer vision, and their object representations across layers match coarsely with visual cortical areas in the brain. However, whether these representations exhibit qualitative patterns seen in human perception or brain representations remains unresolved. Here, we recast well-known perceptual and neural phenomena in terms of distance comparisons, and ask whether they are present in feedforward deep neural networks trained for object recognition. Some phenomena were present in randomly initialized networks, such as the global advantage effect, sparseness, and relative size. Many others were present after object recognition training, such as the Thatcher effect, mirror confusion, Weber’s law, relative size, multiple object normalization and correlated sparseness. Yet other phenomena were absent in trained networks, such as 3D shape processing, surface invariance, occlusion, natural parts and the global advantage. These findings indicate sufficient conditions for the emergence of these phenomena in brains and deep networks, and offer clues to the properties that could be incorporated to improve deep networks.

Diverse routes to expertise in facial recognition

“…This success has also prompted psychologists and neuroscientists to evaluate DNNs as candidate models of face processing in the brain (e.g., see refs. 26,27,28 ). This approach is plausible because neural networks were initially inspired by neurophysiology 29 and their evolution continues to be shaped by discoveries in this field 30,31 .…”

Characterizing the object categories two children see and interact within a dense dataset of naturalistic visual experience

“…However, some word learning studies have found that Zipfian frequency distributions can improve learning (Hendrickson & Perfors, 2019) and that adults can leverage knowledge of common objects to bootstrap the meanings of infrequent objects (Kachergis, Yu, & Shiffrin, 2017). Indeed, the skewed distributions found here and in Clerkin et al (2017) are dramatically different than the uniform distributions of categories fed to modern models of visual category learning -which nonetheless appear to mimic many aspects of the visual system (Jacob, Pramod, Katti, & Arun, 2021). Future work that feeds computational models of category learning the same sequence of visual learning environments experienced by children across development may elucidate the set of online learning mechanisms needed to form robust representations from realistic visual inputs.…”

What do we see behind an occluder? Amodal completion of statistical properties in complex objects

“…Yet, they do not generalize well to occluded objects (Rajaei et al, 2019;Zhu et al, 2019). Their representations of occluded displays have recently found not to match with likely completions (Jacob et al, 2021). However, none of these studies have compared dissimilarities within a given display type as done in our study.…”

“…There have been attempts to train deep networks to be invariant to object occlusions (Tang et al, 2018;Kang and Druckmann, 2020;Kortylewski et al, 2020) though it is not clear if these occlusion-invariant networks would match better with likely more than unlikely completions. We propose that our comparison of occluded with likely and unlikely completions can be added to the list of qualitative differences between deep networks and brains as reported previously (Jacob et al, 2021).…”