The standard theoretical understanding of the lattice thermal conductivity, κ ℓ , of semiconductor alloys assumes that mass disorder is the most important source of phonon scattering. In contrast, we show that the hitherto neglected contribution of force-constant (IFC) disorder is essential to accurately predict the κ ℓ of those polar compounds characterized by a complex atomic-scale structure. We have developed an ab initio method based on special quasirandom structures and Green's functions, and including the role of IFC disorder, and applied it in order to calculate the κ ℓ of In1−xGaxAs and Si1−xGex alloys. We show that, while for Si1−xGex, phonon-alloy scattering is dominated by mass disorder, for In1−xGaxAs, the inclusion of IFC disorder is fundamental to accurately reproduce the experimentally observed κ ℓ . As the presence of a complex atomic-scale structure is common to most III-V and II-VI random semiconductor alloys, we expect our method to be suitable for a wide class of materials.