Phylogenetic relationships among extinct hominoids (apes and humans) are controversial due to pervasive homoplasy and the incompleteness of the fossil record. The bony labyrinth might contribute to this debate, as it displays strong phylogenetic signal among other mammals. However, the potential of the vestibular apparatus for phylogenetic reconstruction among fossil apes remains understudied. Here we test and quantify the phylogenetic signal embedded in the vestibular morphology of extant anthropoids (monkeys, apes and humans) and two extinct apes (Oreopithecus and Australopithecus) as captured by a deformation-based 3D geometric morphometric analysis. We also reconstruct the ancestral morphology of various hominoid clades based on phylogenetically-informed maximum likelihood methods. Besides revealing strong phylogenetic signal in the vestibule and enabling the proposal of potential synapomorphies for various hominoid clades, our results confirm the relevance of vestibular morphology for addressing the controversial phylogenetic relationships of fossil apes.

Late Miocene great apes are key to reconstructing the ancestral morphotype from which earliest hominins evolved. Despite consensus that the late Miocene dryopith great apes Hispanopithecus laietanus (Spain) and Rudapithecus hungaricus (Hungary) are closely related (Hominidae), ongoing debate on their phylogenetic relationships with extant apes (stem hominids, hominines, or pongines) complicates our understanding of great ape and human evolution. To clarify this question, we rely on the morphology of the inner ear semicircular canals, which has been shown to be phylogenetically informative. Based on microcomputed tomography scans, we describe the vestibular morphology of Hispanopithecus and Rudapithecus, and compare them with extant hominoids using landmark-free deformation-based three-dimensional geometric morphometric analyses. We also provide critical evidence about the evolutionary patterns of the vestibular apparatus in living and fossil hominoids under different phylogenetic assumptions for dryopiths. Our results are consistent with the distinction of Rudapithecus and Hispanopithecus at the genus rank, and further support their allocation to the Hominidae based on their derived semicircular canal volumetric proportions. Compared with extant hominids, the vestibular morphology of Hispanopithecus and Rudapithecus most closely resembles that of African apes, and differs from the derived condition of orangutans. However, the vestibular morphologies reconstructed for the last common ancestors of dryopiths, crown hominines, and crown hominids are very similar, indicating that hominines are plesiomorphic in this regard. Therefore, our results do not conclusively favor a hominine or stem hominid status for the investigated dryopiths.

Low bone/matrix intensity difference with X-ray μCT scans in IPS58443.1 is due to the extreme similarity in chemical composition between the matrix and the fossilized tissues, and the presence of high-density elements. In IPS1724, it is attributable to the convergence of enamel and dentine compositions during fossilization. On the contrary, neutron radiation returns very different contrasts for different isotopes of the same element and easily penetrates most metals. Neutron-based μCT scans therefore enable a correct definition of the bone/sediment and enamel/dentine interfaces, and hence a better segmentation of the images stack. We conclude that neutron radiation represents a successful alternative for high-resolution µCT of small-sized fossils that are problematic with X-rays.