Harmonizing the object recognition strategies of deep neural networks with humans

Abstract

The many successes of deep neural networks (DNNs) over the past decade have largely been driven by computational scale rather than insights from biological intelligence. Here, we explore if these trends have also carried concomitant improvements in explaining the visual strategies humans rely on for object recognition. We do this by comparing two related but distinct properties of visual strategies in humans and DNNs: where they believe important visual features are in images and how they use those features to categorize objects. Across 84 different DNNs trained on ImageNet and three independent datasets measuring the where and the how of human visual strategies for object recognition on those images, we find a systematic trade-off between DNN categorization accuracy and alignment with human visual strategies for object recognition. State-of-the-art DNNs are progressively becoming less aligned with humans as their accuracy improves. We rectify this growing issue with our neural harmonizer: a general-purpose training routine that both aligns DNN and human visual strategies and improves categorization accuracy. Our work represents the first demonstration that the scaling laws [1-3] that are guiding the design of DNNs today have also produced worse models of human vision. We release our code and data at https://serre-lab.github.io/Harmonization to help the field build more human-like DNNs.

The best game in town: The reemergence of the language-of-thought hypothesis across the cognitive sciences

“…Nonaka, Majima, Aoki, & Kamitani (2021) thus developed a "Brain Hierarchy Score" that measures similarities between hierarchical structures, applied it to 29 DNNs, and found a negative correlation between image classification performance and similarity to human vision. This finding provides a striking illustration of how DNNs can excel in performance while veering apart from human competence (see also Fel, Felipe, Linsley, & Serre 2022).…”

Alignment with human representations supports robust few-shot learning

“…Several recent studies have also attempted to identify what design choices lead to improved representational alignment in models (Kumar et al, 2022;Muttenthaler et al, 2022;Fel et al, 2022), although Moschella et al (2022) found that even with variation in design choices, many models trained on the same dataset end up learning similar 'relative representations' (embeddings projected into a relational form like a similarity matrix), or in other words, converge to the same representational space. Tucker et al (2022) showed that representational alignment emerges not only in static settings like image classification, but also dynamic reinforcement learning tasks involving human-robot interaction.…”

“…We define representational alignment as the degree to which the latent representations of a model match the latent representations of humans for the same set of stimuli, and refer to models that are representationally aligned with humans as being "human-aligned." Several recent papers have proposed ways to measure (Marjieh et al, 2022a;, explain (Muttenthaler et al, 2022;Kumar et al, 2022), and even improve (Peterson et al, 2018;Fel et al, 2022) the representational alignment of models. However, many models that score low on these alignment metrics still have high performance on downstream tasks like image classification (Kumar et al, 2022;Muttenthaler et al, 2022;Fel et al, 2022).…”

“…Several recent papers have proposed ways to measure (Marjieh et al, 2022a;, explain (Muttenthaler et al, 2022;Kumar et al, 2022), and even improve (Peterson et al, 2018;Fel et al, 2022) the representational alignment of models. However, many models that score low on these alignment metrics still have high performance on downstream tasks like image classification (Kumar et al, 2022;Muttenthaler et al, 2022;Fel et al, 2022).…”

Bird song comparison using deep learning trained from avian perceptual judgments

“…While not yet widely used in bird song research, such methods reduce the dependence on engineering good features, by using spectrograms or waveforms directly as input. Deep learning can achieve impressive accuracy on various tasks, but this neither implies nor demands that their recognition strategies are similar to those of animals (Fel et al, 2022). All comparison methods make assumptions about what constitutes similarity in acoustic signals.…”