An image reconstruction framework for characterizing initial visual encoding

Abstract

We developed an image-computable observer model of the early visual system that operates on fully naturalistic input, based on a framework of Bayesian image reconstruction from retinal cone mosaic excitations. Our model extends previous work on ideal observer analysis and the evaluation of performance beyond psychophysical discrimination tasks, takes into account the statistical regularities of our visual environment, and provides a unifying framework for answering a wide range of questions regarding early vision. Using the error in the reconstruction as a metric, we analyzed the variations of the number of different photoreceptor types on human retina as an optimal design problem. In addition, the reconstructions allow both visualization and quantification of information loss due to physiological optics and cone mosaic sampling, and how these vary with eccentricity. Furthermore, in simulations of color deficiencies and interferometric experiments, we found that the reconstructed images provide a reasonable proxy for directly modeling subjects' percepts. Lastly, we used the reconstruction-based observer for the analysis of psychophysical threshold, and found notable interactions between spatial frequency and chromatic direction in the resulting spatial contrast sensitivity function. Our method should be widely applicable to many experiments and practical applications in which early vision plays an important role.