The structure of the genome, including the architecture, number, and size of its chromosomes, shapes the distribution of genetic diversity and sequence divergence. Importantly, smaller chromosomes experience higher recombination rates than larger ones. To investigate how the relationship between chromosome size and recombination rate affects sequence divergence between species, we adopted an integrative approach that combines empirical analyses and evolutionary simulations. We estimated pairwise sequence divergence among 15 species from three different Mammalian clades - Peromyscus rodents, Mus mice, and great apes - from chromosome-level genome assemblies. We found a strong significant negative correlation between chromosome size and sequence divergence in all species comparisons within the Peromyscus and great apes clades, but not the Mus clade, demonstrating that the dramatic chromosomal rearrangements among Mus species masked the ancestral genomic landscape of divergence in many comparisons. Moreover, our evolutionary simulations showed that the main factor determining differences in divergence among chromosomes of different size is the interplay of recombination rate and selection, with greater variation in larger populations than in smaller ones. In ancestral populations, shorter chromosomes harbor greater nucleotide diversity. As ancestral populations diverge and eventually speciate, diversity present at the onset of the split contributes to greater sequence divergence in shorter chromosomes among daughter species. The combination of empirical data and evolutionary simulations also revealed other factors that affect the relationship between chromosome size and divergence, including chromosomal rearrangements, demography, and divergence times, and deepen our understanding of the role of genome structure on the evolution of species divergence.