Visual perception as retrospective Bayesian decoding from high- to low-level features

Abstract

When a stimulus is presented, its encoding is known to progress from low-to high-level features. How these features are decoded to produce perception is less clear, and most models assume that decoding follows the same low-to high-level hierarchy of encoding. There are also theories arguing for global precedence, reversed hierarchy, or bidirectional processing, but they are descriptive without quantitative comparison with human perception. Moreover, observers often inspect different parts of a scene sequentially to form overall perception, suggesting that perceptual decoding requires working memory, yet few models consider how working-memory properties may affect decoding hierarchy. We probed decoding hierarchy by comparing absolute judgments of single orientations and relative/ordinal judgments between two sequentially presented orientations. We found that lower-level, absolute judgments failed to account for higher-level, relative/ordinal judgments. However, when ordinal judgment was used to retrospectively decode memory representations of absolute orientations, striking aspects of absolute judgments, including the correlation and forward/backward aftereffects between two reported orientations in a trial, were explained. We propose that the brain prioritizes decoding of higher-level features because they are more behaviorally relevant, and more invariant and categorical, and thus easier to specify and maintain in noisy working memory, and that more reliable higher-level decoding constrains less reliable lower-level decoding.Bayesian prior | interreport correlation | bidirectional tilt aftereffect | efficient coding | adaptation theory V isual stimuli evoke neuronal responses (a process termed encoding), which lead to our perceptual estimation of the stimuli (decoding). Experimental studies have firmly established that encoding is hierarchical, progressing from lower-level representations of simpler and less invariant features to higher-level representations of more complex and invariant features along visual pathways (1). Researchers have also studied decoding by using models to relate neuronal responses to perceptual estimation. Most models posit, explicitly or implicitly, that decoding follows the same low-to high-level hierarchy, often in the form of what we call the absolute-to-relative assumption (2-6). For example, these models may decode V1 responses to a line into a perceived orientation of, say 51.2°(or a distribution around it). Psychophysically, this is termed an absolute judgment. To determine the relationship between two lines, the models first decode each orientation separately and the two resulting absolute orientations are then compared. For instance, the two absolute orientations may be subtracted to obtain the angle between the lines (relative orientation), or the sign of the difference may be used to determine whether the second line is clockwise or counterclockwise from the first (ordinal orientation discrimination). Absolute orientation of a single line is a simpler, less invariant, lower-...

Benefits of commitment in hierarchical inference

“…These biases can be thought of as a form of consistency (Brehm, 1956) or confirmation bias (Nickerson, 1998) where the perceptual estimate aligns with and confirms the chosen category (Bronfman et al, 2015; Talluri et al, 2018). Importantly, these biases are not dependent on subjects making an explicit, overt choice of a high-level interpretation; similar biases were observed in tasks that did not require an explicit categorical choice (Wu et al, 2009; Zamboni et al, 2016; Ding et al, 2017), indicating that committing to a high-level interpretation could be an inherent inference strategy. As proposed (Stocker and Simoncelli, 2007), and refined and validated more recently (Luu and Stocker, 2018), the behavioral biases in above examples are remarkably well described by a conditioned observer model .…”

“…Since first proposed (Stocker and Simoncelli, 2007), data from a growing number of psychophysical studies suggest that humans perform conditioned inference in certain situations (Jazayeri and Movshon, 2007; Zamboni et al, 2016; Wu et al, 2009; Ding et al, 2017; Fritsche and de Lange, 2019; Luu and Stocker, 2018); i.e. , they commit to a single high-level interpretation rather than taking into account all possible high-level interpretations when performing low-level feature inference.…”

“…Specifically, it suggests that the commitment to a high-level interpretation can be seen as creating a robust summary representation of sensory evidence, which then, at the time of recall, is used in order to improve inference accuracy. In fact, conditioned inference can predict some of the memory biases that have been reported in the recall of color (Bae et al, 2015) or other low-level visual features (Ding et al, 2017; Luu et al, 2017). It provides a normative motivation for memory biases and potentially other similar forms of confirmation biases (Talluri et al, 2018; Lange et al, 2019).…”

High- to low-level decoding does not generally improve perceptual performance

“… Ding et al (1) recently proposed that the brain automatically encodes high-level, relative stimulus information (i.e. the ordinal relation between two lines), which it then uses to constrain the decoding of low-level, absolute stimulus features (i.e.…”

“…This is an interesting idea that is in line with the self-consistent Bayesian observer model (2, 3) and may have important implications for understanding how the brain processes sensory information. However, the notion suggested in Ding et al (1) that the brain uses this decoding strategy because it improves perceptual performance is misleading. Here we clarify the decoding model and compare its perceptual performance under various noise and signal conditions.…”