Species from diverse animal lineages, including molluscs, hemichordates, and chordates, have retained groups of orthologous genes on the same chromosomes for over half a billion years since the last common ancestor of bilaterians. Though there are notable exceptions, it has been proposed that the conservation of chromosome-scale gene linkages is the norm among animals. Here, by examining interchromosomal rearrangements in 64 chromosome-level genomes across 15 bilaterian phyla and at least 52 classes, we show that cases of genome structure conservation are exceptionally rare. Large-scale genome restructuring events correlate with increased rates of protein sequence evolution and may represent a previously underappreciated contributor to adaptation and animal diversity.