Reconstruction of natural images from responses of primate retinal ganglion cells

Abstract

The visual message conveyed by a retinal ganglion cell (RGC) is often summarized by its spatial receptive field, but in principle should also depend on other cells' responses and natural image statistics. To test this idea, linear reconstruction (decoding) of natural images was performed using combinations of responses of four high-density macaque RGC types, revealing consistent visual representations across retinas. Each cell's visual message, defined by the optimal reconstruction filter, reflected natural image statistics, and resembled the receptive field only when nearby, same-type cells were included. Reconstruction from each cell type revealed different and largely independent visual representations, consistent with their distinct properties. Stimulus-independent correlations primarily affected reconstructions from noisy responses. Nonlinear response transformation slightly improved reconstructions with either ON or OFF parasol cells, but not both. Inclusion of ON-OFF interactions enhanced reconstruction by emphasizing oriented edges, consistent with linear-nonlinear encoding models. Spatiotemporal reconstructions revealed similar spatial visual messages.

Nonlinear decoding of natural images from large-scale primate retinal ganglion recordings

“…These linear filters eventually allowed for a sparse mapping between RGC units and image pixels so that only the most informative units for each pixel would be used as inputs for the nonlinear decoder [25]. Consistent with previous findings [12], both linear decoders successfully decoded the global features of the stimuli by accurately modeling the low-pass images ( Figure 4).…”

“…For high-pass decoding, the neural network decoder exhibited a 1.9% (±0.4) increase in pixel-wise MSE when temporal correlations were removed, while the ridge decoder experienced a 0.04% (±0.07) increase in MSE ( Figure 6, Bottom); i.e., nonlinear high-pass decoding is dependent on temporal correlations while linear high-pass decoding is not. Removing cross-neuronal correlations yielded no significant changes in either decoder, consistent with [12]. Meanwhile, for low-pass decoding, both decoders were equally and significantly affected by removing temporal correlations, as indicated by the 17.5% (±6.7) and 14.2% (±8.9) increases in MSE for the neural network and linear decoders, respectively (Figure 6, Bottom).…”

“…for linearly decoding the low-pass images. Both decoders only considered the neural The LASSO spatial filters were roughly similar in appearance to the corresponding RGC unit receptive fields calculated from spike-triggered averages of white noise recordings (data not shown; see [12]). These linear filters eventually allowed for a sparse mapping between RGC units and image pixels so that only the most informative units for each pixel would be used as inputs for the nonlinear decoder [25].…”

“…All decoding results were obtained on retinal datasets consisting of macaque RGC spike responses to natural scene images [12]. Two identically prepared datasets, each containing responses to 10,000 images, were used for independent validation of our decoding methods.…”

“…The retina has long provided a useful testbed for decoding methods, since mapping retinal ganglion cell (RGC) responses into a decoded image provides a direct visualization of decoding model performance. Most approaches to decoding images from RGCs have depended on linear methods, due to their interpretability and computational efficiency [1,11,12]. Although linear methods successfully decoded spatially uniform white noise stimuli [1] and the coarse structure of natural scene stimuli from RGC population responses [12], they largely fail to recover finer visual details of naturalistic images.…”