Across a species range, spatially-varying environments can drive the evolution of local adaptation. Multiples sources of environmental heterogeneity, at small and large scales, draw complex landscapes of selection which may challenge adaptation, particularly when gene flow is high. Because linkage opposes gene flow but also limits the efficiency of natural selection by contrasting pressures, the key to multidimensional adaptation may reside in the heterogeneity of recombination along the genome. Structural variants like chromosomal inversions are important recombination modifiers that form massive co-segregating genomic blocks linking together alleles at numerous genes. In this study, we investigate the influence of chromosomal rearrangements on genetic variation to ask how their contribution to adaptation with gene flow varies across geographic scales. We sampled the seaweed fly Coelopa frigida along a bioclimatic gradient of 10° of latitude, a salinity gradient and across a range of heterogeneous, patchy habitats. We assembled a high-quality genome to analyse 1,446 low-coverage whole-genome sequences, and we found large non-recombining genomic regions, including putative inversions. In contrast to the collinear regions depicting extensive gene flow, inversions and low-recombining regions differentiated populations more strongly, either along an ecogeographic cline or at a fine-grained scale. Those genomic regions were disproportionately involved in associations with environmental factors and adaptive phenotypes, albeit with contrasting patterns between the different recombination modifiers. Altogether, our results highlight the importance of recombination in shaping the selection-migration balance and show that a set of several inversions behave as modular cassettes facilitating adaptation to environmental heterogeneity at local and large scales.