See an object, hear an object file: Object correspondence transcends sensory modality

Abstract

An important task of perceptual processing is to parse incoming information into distinct units and to keep track of those units over time as the same, persisting representations. Within the study of visual perception, maintaining such persisting object representations is helped by ''object files''*episodic representations that store (and update) information about objects' properties and track objects over time and motion via spatiotemporal information. Although object files are typically discussed as visual, here we demonstrate that objectÁfile correspondence can be computed across sensory modalities. An object file can be initially formed with visual input and later accessed with corresponding auditory information, suggesting that object files may be able to operate at a multimodal level of perceptual processing.Perception is a complicated process with impressive feats seemingly accomplished with ease. For example, immense amounts of undifferentiated information are smoothly transformed into meaningful units; photons of

Predicting sights from sounds: 6-month-olds’ intermodal numerical abilities

“…Although the Approximate Number System is capable of representing very small numerosities such as 1, 2, or 3 (Cordes & Brannon, 2009a; Cordes, Gelman, Gallistel, & Whalen, 2001), much evidence suggests that infants often represent arrays containing 1, 2, or 3 items in terms of separate individuals rather than as an array with an approximate cardinal value (Feigenson & Carey, 2003; Feigenson, Carey, & Hauser, 2002; Feigenson et al, 2004; Xu, 2003). This would be akin to representing an array containing 3 dots as Object A , Object B , Object C , rather than as “exactly 3” or “approximately 3.” For this reason the above studies are consistent with infants having determined intermodal matches between individual objects and individual sounds or tactile experiences (Jordan, Clark, & Mitroff, 2010), without invoking any numerical representations at all.…”

Grades of Multisensory Awareness

“…For example, Cinel et al (, pp. 1244–1245) say, ‘These results demonstrate that ICs [Illusory Conjunctions] are possible not only within the visual modality but also between two different modalities: vision and touch,’ and conclude, ‘[I]nformation converges preattentively for binding from different sensory modalities … this binding process is modulated by the parietal lobe.’ Jordan et al (, p. 501) report ‘a standard, robust OSPB [Object Specific Preview Benefit]’ between vision and audition and say their data ‘explicitly demonstrate object files can operate across visual and auditory modalities.’ Zmigrod et al (, pp. 682–683) support ‘episodic multimodal representations’ rather than mere intermodal interactions and conclude that feature binding occurs across modalities.…”

Perceiving expressions of emotion: What evidence could bear on questions about perceptual experience of mental states?

“…e operation of object indexes may turn out not to be specifically visual, as Jordan et al 2010's finding that acoustic stimuli can cause object indexes to be updated hints. e perverse restriction allows us to consider research on object indexes even if this is so.…”