We study the phase transition from a topological to a normal insulator with concentration x in (Bi1−xInx)2Se3 and (Bi1−xSbx)2Se3 in the Bi2Se3 crystal structure. We carry out first-principles calculations on small supercells, using this information to build Wannierized effective Hamiltonians for a more realistic treatment of disorder. Despite the fact that the spin-orbit coupling (SOC) strength is similar in In and Sb, we find that the critical concentration xc is much smaller in (Bi1−xInx)2Se3 than in (Bi1−xSbx)2Se3. For example, the direct supercell calculations suggest that xc is below 12.5% and above 87.5% for the two alloys respectively. More accurate results are obtained from realistic disordered calculations, where the topological properties of the disordered systems are understood from a statistical point of view. Based on these calculations, xc is around 17% for (Bi1−xInx)2Se3, but as high as 78%-83% for (Bi1−xSbx)2Se3. In (Bi1−xSbx)2Se3, we find that the phase transition is dominated by the decrease of SOC, with a crossover or "critical plateau" observed from around 78% to 83%. On the other hand, for (Bi1−xInx)2Se3, the In 5s orbitals suppress the topological band inversion at low impurity concentration, therefore accelerating the phase transition. In (Bi1−xInx)2Se3 we also find a tendency of In atoms to segregate.