Humans are endowed with an intuitive number sense that allows us to perceive and estimate numerosity without relying on language. It is controversial, however, as to whether there is a neural mechanism for direct perception of numerosity or whether numerosity is perceived indirectly via other perceptual properties. In this study, we used a novel regression-based analytic method, which allowed an assessment of the unique contributions of visual properties, including numerosity, to explain visual evoked potentials of participants passively viewing dot arrays. We found that the human brain is uniquely sensitive to numerosity and more sensitive to changes in numerosity than to changes in other visual properties, starting extremely early in the visual stream: 75 ms over a medial occipital site and 180 ms over bilateral occipitoparietal sites. These findings provide strong evidence for the existence of a neural mechanism for rapidly and directly extracting numerosity information in the human visual pathway.