Millimeter-wave (mmWave) systems have been considered as a promising candidate for 5G networks because of their potential advances in significant bandwidth enhancement. However, due to the extremely high operating frequency of mmWave systems, they generally suffer from severe frequencyselective propagation and nonlinear distortion in the power amplifier, which introduces unfavorable impact on the signal detection process. We discover that for indoor mmWave communications, the constellation of signals becomes much more ''clean and tidy'' at the receiver side compared with the current wireless systems (e.g., LTE and WiFi), thanks to the channel sparsity characteristic of mmWave communications. Motivated by this observation, we propose in this paper several detection algorithms. Specifically, K-means clustering (KMC) algorithm is first introduced into clustering signal detection due to its advantage in the circle or spherical cluster shape. Then, an improved KMC detector is proposed to avoid the deficiencies of KMC for the error floor and high complexity. Moreover, a density and distance-based clustering detector, a non-iterative algorithm, is proposed and it does not need to preset the number of clusters. The aboveproposed algorithms do not require any prior information about the power amplifier and the channel state information at the receiver end, which presents noticeable practical achievements on cost, complexity, and hardware constraints. The simulation results verify the effectiveness of the proposed schemes.INDEX TERMS Millimeter-wave (mmWave) communication, nonlinear signal detection, nonlinear power amplifier, unsupervised clustering.