Through a Dog's Eyes: fMRI Decoding of Naturalistic Videos from the Dog Cortex

Abstract

Recent advancements using machine learning and functional magnetic resonance imaging (fMRI) to decode visual stimuli from the human and nonhuman cortex have resulted in new insights into the nature of perception. However, this approach has yet to be applied substantially to animals other than primates, raising questions about the nature of such representations across the animal kingdom. Here, we used awake fMRI in two domestic dogs and two humans, obtained while each watched specially created dog-appropriate naturalistic videos. We then trained a neural net (Ivis) to classify the video content from a total of 90 min of recorded brain activity from each. We tested both an object-based classifier, attempting to discriminate categories such as dog, human, and car, and an action-based classifier, attempting to discriminate categories such as eating, sniffing, and talking. Compared to the two human subjects, for whom both types of classifier performed well above chance, only action-based classifiers were successful in decoding video content from the dogs. These results demonstrate the first known application of machine learning to decode naturalistic videos from the brain of a carnivore and suggest that the dog's-eye view of the world may be quite different from our own.

Domestic dogs as a comparative model for social neuroscience: Advances and challenges

“…The majority of the fMRI studies exploring the neural bases of agent perception focused specifically on face processing (Bunford et al, 2020;Cuaya et al, 2016;Dilks et al, 2015;Gillette et al, 2022;Hernández-Pérez et al, 2018;Szabó et al, 2020;Thompkins et al, 2018) with two studies, additionally investigating body perception (Boch et al 2023a; and functional localizer of Boch et al 2023b). All studies show converging evidence that the occipito-temporal lobe plays a key role in agent perception in dogs, which is in line with similar findings in humans, suggesting a Studies so far showed that these agent-responsive areas result in greater activation for faces compared to inanimate objects (Boch et al 2023a;Boch et al 2023b;Cuaya et al, 2016;Dilks et al, 2015;Gillette et al, 2022) and scenes (Dilks et al, 2015), but not all found greater activation to scrambled controls (Dilks et al, 2015;Szabó et al, 2020; but see Boch et al, 2023a;Boch et al, 2023b). The ectomarginal, mid and caudal suprasylvian agent gyrus were also more active during perception of bodies than inanimate objects and scrambled controls (Boch et al 2023a;Boch et al 2023b) but seem to be largely involved in the perception of both faces and bodies, except for a patch in the suprasylvian gyrus, which resulted in greater activation for bodies compared to faces.…”

“…The ectomarginal, mid and caudal suprasylvian agent gyrus were also more active during perception of bodies than inanimate objects and scrambled controls (Boch et al 2023a;Boch et al 2023b) but seem to be largely involved in the perception of both faces and bodies, except for a patch in the suprasylvian gyrus, which resulted in greater activation for bodies compared to faces. Viewing bodies compared to faces or inanimate and scrambled controls also resulted in greater task-based functional connectivity between the primary visual cortex (V1) and the caudal suprasylvian agent-responsive area (Boch Functional MRI studies so far localized neocortical areas involved in agent (i.e., faces and bodies; see e.g., Boch et al 2023a;Dilks et al, 2015;Szabó et al, 2020) and action (see e.g., Phillips et al, 2022) perception and showed that these areas exchange information (i.e., taskbased functional connectivity) with primary visual cortex (V1) during face, body and action perception (Boch et al 2023b). First evidence suggests that areas in the multisensory sylvian gyrus process dynamic social aspects of visual social cues (e.g., emotion perception or social interactions; Boch et al 2023b;Hernández-Pérez et al, 2018;Karl et al, 2021;Phillips et al, 2022) and sensitivity for species identity in the ectomarginal and mid suprasylvian gyrus (Boch et al 2023b;Bunford et al, 2020).…”

“…Viewing bodies compared to faces or inanimate and scrambled controls also resulted in greater task-based functional connectivity between the primary visual cortex (V1) and the caudal suprasylvian agent-responsive area (Boch Functional MRI studies so far localized neocortical areas involved in agent (i.e., faces and bodies; see e.g., Boch et al 2023a;Dilks et al, 2015;Szabó et al, 2020) and action (see e.g., Phillips et al, 2022) perception and showed that these areas exchange information (i.e., taskbased functional connectivity) with primary visual cortex (V1) during face, body and action perception (Boch et al 2023b). First evidence suggests that areas in the multisensory sylvian gyrus process dynamic social aspects of visual social cues (e.g., emotion perception or social interactions; Boch et al 2023b;Hernández-Pérez et al, 2018;Karl et al, 2021;Phillips et al, 2022) and sensitivity for species identity in the ectomarginal and mid suprasylvian gyrus (Boch et al 2023b;Bunford et al, 2020). However, investigations into the specific roles of the localized agent and action areas remain to be undertaken.…”

Functional mapping of the somatosensory cortex using noninvasive fMRI and touch in awake dogs

“…To understand how higher order cognition is processed in the dog´s brain, we also need to understand the brain´s fundamental organization and how it processes sensory input at the lower levels. To this end, recent research has mapped out the visual, olfactory and auditory cortices in awake and unrestrained dogs using functional magnetic resonance imaging (fMRI; Andics et al 2014 , 2016 ; Boch et al 2021 , 2023 ; Bunford et al 2020 ; Cuaya et al 2016 , 2022 ; Dilks et al 2015 ; Gillette et al 2022 ; Jia et al 2014 ; Phillips et al 2022 ). However, our best understanding of the canine somatosensory cortex dates back to almost 70 years ago (Fritsch & Hitzig, 1870/ 1963 ; Hamuy et al 1956 ), using invasive methods, a small sample and focusing on selected parts of the canine cortex.…”

Functional mapping of the somatosensory cortex using noninvasive fMRI and touch in awake dogs

“…To understand how higher order cognition is processed in the dog´s brain, we also need to understand the brain´s fundamental organization and how it processes sensory input at the lower levels. To this end, recent research has mapped out the visual, olfactory and auditory cortices in awake and unrestrained dogs using functional magnetic resonance imaging (fMRI; Andics et al, 2014Andics et al, , 2016Boch et al, 2021Boch et al, , 2023Bunford et al, 2020;Cuaya et al, 2016Cuaya et al, , 2022Dilks et al, 2015;Gillette et al, 2022;Jia et al, 2014;Phillips et al, 2022). However, our best understanding of the canine somatosensory cortex dates back to almost 70 years ago (Hamuy et al, 1956), using invasive methods, a small sample and focusing on selected parts of the canine cortex.…”