The presence of male-female brain differences has long been a controversial topic. Yet simply negating the existence of biological differences has detrimental consequences for all sexes and genders, particularly for the development of accurate diagnostic tools, effective drugs and understanding of disease. The most well-established morphological difference is size, with males having on average a larger brain than females; yet a mechanistic understanding of how this difference arises remains to be elucidated. Here, we use brain organoids to test the roles of sex chromosomes and sex steroids during development. While we show no observable differences between XX and XY brain organoids, sex steroids, namely androgens, increase proliferation of cortical neural progenitors. Transcriptomic analysis reveals effects on chromatin remodelling and HDAC activity, both of which are also implicated in the male-biased conditions autism spectrum disorder and schizophrenia. Finally, we show that higher numbers of progenitors result specifically in increased upper-layer excitatory neurons. These findings uncover a hitherto unknown role for male sex hormones in regulating excitatory neuron number within the human neocortex and represent a first step towards understanding the origin of human sex-related brain differences.