It has been challenging to separate the time-frequency (TF) overlapped wireless communication signals with unknown number of sources in underdetermined cases. To address this issue, a novel blind separation strategy based on a TF soft mask is proposed in this paper. Based on the clustering property of the signals in the sparse domain, the angular probability density distribution is obtained by the kernel density estimation (KDE) algorithm, and then the number of source signals is identified by detecting the peak points of the distribution. Afterwards, the contribution degree function is designed according to the cosine distance to calculate the contribution degrees of the source signals in the mixed signals. The separation of the TF overlapped signals is achieved by constructing a soft mask matrix based on the contribution degrees. In this paper, the simulations are performed with digital signals of the same modulation and different modulation respectively. The results show that the proposed algorithm has better anti-aliasing and anti-noise performance than the benchmark algorithms.