Phase diagram calculations are performed for incoherent and coherent mixtures of an [Formula: see text] (InAsSb) ternary alloy, which is an important material for the applications to infrared detector technology. Our calculations are based on the cluster expansion approach and Monte Carlo simulations combined with first-principles total energy calculations in the framework of density functional theory with Perdew–Burke–Ernzerhof (PBE) and Heyd–Scuseria–Ernzerhof (HSE) exchange-correlation functionals. Because of a lattice mismatch ([Formula: see text]7%) between InAs and InSb, coherency strain plays an important role for the phase stability of the InAsSb alloys. The alloys without the coherency strain (incoherent mixtures) show a miscibility gap with the critical temperature at [Formula: see text]700 K with 42% (45%) Sb concentration in PBE (HSE), which is in good agreement with the experimentally determined equilibrium miscibility gap temperature. The alloys with the coherency strain (coherent mixtures) show several ground states whose structures are short period superlattices along the [201] direction. The critical temperature is [Formula: see text]200 K with 50% Sb concentration in both PBE and HSE, which is reduced by [Formula: see text]500 K compared to that of incoherent mixtures. This reduction of the critical temperature is consistent with the experimental observation where the homogeneous InAsSb alloy continues to grow inside the empirical miscibility gap.