Human visual cortex is organized with striking consistency across individuals. While recent findings demonstrate an unexpected coupling between functional and cytoarchitectonic regions relative to the folding of human visual cortex, a unifying principle linking these anatomical and functional features of cortex remains elusive. To fill this gap in knowledge, we combined independent and ground truth measurements of human cytoarchitectonic regions and genetic tissue characterization within the visual processing hierarchy. Using a data-driven approach, we examined if differential gene expression among cortical areas could explain the organization of the visual processing hierarchy into early, middle, and late processing stages. This approach revealed that the visual processing hierarchy is explained by two opposing gene expression gradients: one that contains a series of genes with expression magnitudes that ascend from the first processing stage (e.g. area hOc1, or V1) to the last processing stage (e.g. area FG4) and another that contains a separate series of genes that show a descending gradient. In the living human brain, each of these gradients correlates strongly with anatomical variations along the visual hierarchy such as the thickness or myelination of cortex. We further reveal that these genetic gradients emerge along unique trajectories in human development: the ascending gradient is present at 10-12 gestational weeks, while the descendent gradient emerges later (19-24 gestational weeks). Interestingly, it is not until early childhood (before 5 years of age) that the two expression gradients achieve their adult-like mean expression values. Finally, additional analyses in non-human primates (NHP) reveal the surprising finding that only the ascending, but not the descending, expression gradient is evolutionarily conserved. These findings create one of the first models bridging macroscopic features of human cytoarchitectonic areas in visual cortex with microscopic features of cellular organization and genetic expression, revealing that the hierarchy of human visual cortex, its cortical folding, and the cytoarchitecture underlying its computations, can be described by a sparse subset (~200) of genes, roughly one-third of which are not shared with NHP. These findings help pinpoint the genes contributing to both healthy cortical development and the cortical biology distinguishing humans from other primates, establishing essential groundwork for understanding future work linking genetic mutations with the function and development of the human brain.