Visual performance varies around the visual field. It is best near the fovea compared to the periphery, and at iso-eccentric locations it is best on the horizontal, intermediate on the lower, and poorest on the upper meridian. The fovea-to-periphery performance decline is linked to the decreases in cone density, retinal ganglion cell (RGC) density, and V1 cortical magnification factor (CMF) as eccentricity increases. The origins of radial asymmetries are not well understood. Optical quality and cone density vary across the retina, but recent computational modeling has shown that these factors can only account for a small percentage of behavior. Here, we investigate how visual processing beyond the cones contributes to radial asymmetries in performance. First, we quantify the extent of asymmetries in cone density, midget RGC density, and V1-V2 CMF. We find that both radial asymmetries and eccentricity gradients are amplified from cones to mRGCs, and from mRGCs to cortex. Second, we extend our previously published computational observer model to quantify the contribution of spatial filtering by mRGCs to behavioral asymmetries. Starting with photons emitted by a visual display, the model simulates the effect of human optics, fixational eye movements, cone isomerizations and mRGC spatial filtering. The model performs a forced choice orientation discrimination task on mRGC responses using a linear support vector machine classifier. The model shows radial asymmetries in performance that are larger than those from a model operating on the cone outputs, but considerably smaller than those observed from human performance. We conclude that spatial filtering properties of mRGCs contribute to radial performance differences, but that a full account of these differences will entail a large contribution from cortical representations.