Many animals use visual numerosity, the number of items in a group, to guide behavior. Neurons in human association cortices show numerosity-tuned responses, decreasing amplitude with distance from a specific numerosity. How are such responses derived from early visual responses? Recent studies show aggregate response amplitudes in human early visual cortex monotonically increase with numerosity, regardless of object size and spacing. This is surprising because numerosity is typically considered a high-level visual or cognitive feature. Here we first use computational modelling of 7T fMRI data to show these monotonic responses originate at the stimulus's retinotopic location in primary visual cortex (V1). Given this location, we then ask whether these monotonic responses can be better described by V1's established response properties. We characterize the Fourier decomposition (into contrast at specific orientations and spatial frequencies) of laboratory numerosity stimuli. This demonstrates that aggregate Fourier power (at all orientations and spatial frequencies) nonlinearly follows numerosity with little effect of item size, spacing or shape: it is proportional to numerosity at a fixed contrast. This nonlinear relationship lets us distinguish predictions of responses to Fourier power and numerosity. Monotonic responses are better predicted by Fourier power, later tuned responses are better predicted by numerosity. Tuned responses emerge after lateral occipital cortex and are independent of retinotopic location. We propose that numerosity's straightforward perception and neural responses reflect its straightforward estimation from early visual spatial frequency domain image representations. Our numerical vision may have built on behaviorally beneficial analysis of spatial frequency in simpler animals.