Plants and their specialized flower visitors provide valuable insights into the evolutionary consequences of species interactions. In particular, antagonistic interactions between insects and plants have often been invoked as a major driver of diversification. Here we use a tropical community of palms and their specialized insect flower visitors to understand whether antagonisms lead to higher population divergence. Interactions between the palms Syagrus coronata and Syagrus botryophora and the weevils that visit their flowers range from brood pollination to florivory and commensalism. We use genomics to test the role of insect-host interactions in the early stages of diversification of nine species of beetles associated with these plams by using a model of isolation by environment. We find a surprising number of cryptic species, which in pollinating weevils coexist across a broad geographical range but are always associated with different hosts for non-pollinators. The degree to which insect populations are structured by the genetic divergence of plant populations varies. This variation is uncorrelated with the kind of interaction, showing that, at least in this system, antagonistic interactions are not associated with higher genetic differentiation. It is likely that more general aspects of host use, affecting plant-associated insects regardless of the outcomes of their interactions, are more important drivers of population divergence.