We consider a dilute atomic gas of two species of fermions with unequal concentrations under a Feshbach resonance. We find that the system can have distinct properties due to the unbound fermions. The uniform state is stable only when either (a) beyond a critical coupling strength, where it is a gapless superfluid, or (b) when the coupling strength is sufficiently weak, where it is a normal Fermi gas mixture. Phase transition(s) must therefore occur when the resonance is crossed.PACS numbers: 03.75. Ss, 05.30.Fk, 34.90.+q Feshbach resonance [1] has opened up a new playground for the field of cold trapped atoms. Using this resonance, the effective interaction between the atoms can be varied over a wide range. In particular, for two fermion species with a Feshbach resonance between them, the ground state can be tuned from a weak-coupling Bardeen Cooper Schrieffer (BCS) superfluid to a strong coupling regime where the Fermions pair-up to form Bosons which in turn undergo Bose Einstein Condensation (BEC) [2,3].Though this problem has been under intense theoretical [4] and experimental [5] investigations, almost all works thus far are restricted to the case where the concentrations of the two fermionic species are equal. We here generalize this study to the case of unequal populations of the two species, and investigate in detail the thermodynamic stability of this system, in particular the question when the uniform state can be stable.Studies of fermions with unequal populations or mismatched Fermi surfaces and a pairing interaction have a long history. It was studied by Fulde and Ferrell, Larkin and Ovchinnikov (FFLO) [6] in the 1960's with relation to superconductivity in materials with ferromagnetically coupled paramagnetic impurities. It was found that in this case the system is likely to have an inhomogeneous gapless superconducting phase. Advances in techniques of manipulating dilute ultracold atoms have revived interests in the related problems [7]. These studies, in our present language, are still restricted to the weak-coupling regime. We, however, would extend our analysis to all coupling strengths.In the "canonical" problem of two species of fermions with equal mass (say, spin up and spin down electrons) and equal concentrations (thus a single Fermi surface), if the cross-species interaction is varied from weak to strong coupling, at low temperatures the system would undergo a smooth crossover from a superfluid with loosely bound Cooper pairs (the "BCS" limit) to one with condensation of tightly bound bosonic molecules (the "BEC" limit) [2]. The situation, however, can be very different if one considers two species of fermions with unequal concentrations (i.e. mismatched Fermi surfaces), even if they have identical mass. This can be anticipated because, on the one hand, far into the BCS side, the system is basically in the FFLO regime [7] and therefore must go into a spatially inhomogeneous phase. On the other hand, in the far end of the BEC side, the system is expected to behave like an ordinary (w...
