Attention helps us to select what is relevant from the enormous amounts of information taken up by our senses. However, it remains unclear just how early in sensory processing attentional selection can occur. Here, we investigated this question in healthy volunteers by assessing the effect of attentional load on the earliest component (C1) of the visual evoked potential (VEP). We mapped participants’ C1 responses to task-irrelevant peripheral textures of different densities and then selected those textures eliciting maximal C1 amplitudes in the upper and lower visual field in each participant. In a second experimental session, these optimal C1 stimuli served as peripheral distracters while participants performed easy or difficult detection tasks at fixation. Our results show a reduction of C1 amplitudes under high attentional load selectively in the lower visual field. This asymmetric effect is opposite to previously reported results obtained without preselecting stimuli. We conclude that attentional selection during early visual processing is possible, but depends on the interaction between anatomic and functional anisotropies of the visual system. This underscores the importance of precisely delineating when, where, and how attentional filtering can operate on initial perceptual processing.