Neural model of visual stereomatching: slant, transparency and clouds

Abstract

Stereomatching of oblique and transparent surfaces is described using a model of cortical binocular 'tuned' neurons selective for disparities of individual visual features and neurons selective for the position, depth and 3D orientation of local surface patches. The model is based on a simple set of learning rules. In the model, monocular neurons project excitatory connection pathways to binocular neurons at appropriate disparities. Binocular neurons project excitatory connection pathways to appropriately tuned 'surface patch' neurons. The surface patch neurons project reciprocal excitatory connection pathways to the binocular neurons. Anisotropic intralayer inhibitory connection pathways project between neurons with overlapping receptive fields. The model's responses to simulated stereo image pairs depicting a variety of oblique surfaces and transparently overlaid surfaces are presented. For all the surfaces, the model (i) assigns disparity matches and surface patch representations based on global surface coherence and uniqueness, (ii) permits coactivation of neurons representing multiple disparities within the same image location, (iii) represents oblique slanted and tilted surfaces directly, rather than approximating them with a series of frontoparallel steps, (iv) assigns disparities to a cloud of points at random depths, like human observers and unlike Prazdny's (1985) method, and (v) causes globally consistent matches to override greedy local matches. The model represents transparency, unlike the model of Marr and Poggio (1976), and it assigns unique disparities, unlike the model of Prazdny.

Pre–attentive segmentation and correspondence in stereo

“…Salient features include a target of a unique depth in the scene or a depth discontinuity, and are important as an initial step towards segmentation and for attracting visual attention for further processing. I believe that this model is the rst to address computational mechanisms for pre-attentive stereo segmentation in the sense of pop-out or saliency highlights by depth, in addition to solving the stereo correspondence problem that has been extensively studied previously (Marr & Poggio 1976;Prazdny 1985;Pollard et al 1985;Qian & Sejnowski 1989;Marshall et al 1996). Addressing both segmentation and correspondence at the same time is important since there was an apparent con ict at the level of neural mechanism between these two computational goals-segmentation requires mutual inhibition and the correspondence requires mutual excitation between nearby cells that are tuned to similar disparities.…”

“…Note that transparency has been traditionally dif cult to model, since it requires the model to accommodate two discrete depth values at a given visual angle. Although the original model by Marr & Poggio (1976) could not account for it because of its particular implementation of the smoothness constraint, more recent stereo models that solve the correspondence problem (Qian & Sejnowski 1989;Marshall et al 1996) do successfully model it. …”

“…This correspondence problem is ill-posed as there are many possible false matches between individual dots, as illustrated in gure 1 for the wallpaper illusion. These observations led to a substantial research focus on solving the correspondence problem in early stereo vision, as exempli ed by the well-known model by Marr & Poggio (1976, 1979 in the 1970s and many other models since then (Prazdny 1985;Pollard et al 1985;Qian & Sejnowski 1989;Nasrabadi et al 1989;Geiger et al 1995;Marshall et al 1996;McLoughlin & Grossberg 1998;Watanabe & Fukushima 1999;Read 2002). These models employ cooperative algorithms (or indirectly, using optimizations or Bayesian approaches) that use contextual information to nd true binocular matches among all possible matches.…”

“…Intuitively, there would appear to be a con ict. The smoothness constraint requires that nearby cortical cells tuned to similar depths facilitate each other's activities, as implemented in traditional models (Marr & Poggio 1976;Prazdny 1985;Pollard et al 1985;Qian & Sejnowski 1989;Marshall et al 1996). However, such mutual excitation implies that, for a stimulus composed of a single target (e.g.…”