Twenty years ago, Ringo and colleagues proposed that maintaining absolute connectivity in larger compared with smaller brains is computationally inefficient due to increased conduction delays in transcallosal information transfer and expensive with respect to the brain mass needed to establish these additional connections. Therefore, they postulated that larger brains are relatively stronger connected intrahemispherically and smaller brains interhemispherically, resulting in stronger functional lateralization in larger brains. We investigated neuronal interconnections in 138 large and small human brains using diffusion tensor imaging-based fiber tractography. We found a significant interaction between brain size and the type of connectivity. Structural intrahemispheric connectivity is stronger in larger brains, whereas interhemispheric connectivity is only marginally increased in larger compared with smaller brains. Although brain size and gender are confounded, this effect is gender-independent. Additionally, the ratio of interhemispheric to intrahemispheric connectivity correlates inversely with brain size. The hypothesis of neuronal interconnectivity as a function of brain size might account for shorter and more symmetrical interhemispheric transfer times in women and for empirical evidence that visual and auditory processing are stronger lateralized in men. The hypothesis additionally shows that differences in interhemispheric and intrahemispheric connectivity are driven by brain size and not by gender, a finding contradicting a recently published study. Our findings are also compatible with the idea that the more asymmetric a region is, the smaller the density of interhemispheric connections, but the larger the density of intrahemispheric connections. The hypothesis represents an organization principle of the human connectome that might be applied also to non-human animals as suggested by our cross-species comparison.