There are few documents about the mining of the latent abilities of students. Factor analysis can be used to the mining of the latent factors. However, the factor analysis method has the shortcoming that its load matrix is not easy to be explained, which will affect the subsequent clustering results. Therefore, a new method is proposed for mining students' latent abilities and clustering the students. The method relies mainly on the iteration of principal factors, thus overcoming the shortcoming of factor analysis method. It consists of the iterative principal factor algorithm, the least square algorithm and the k-means algorithm. Through these algorithms, students' latent abilities can be recognized, and students can precisely be divided into different clusters. Compared with k-means algorithm and the clustering algorithm based on factor analysis, experiments show that the proposed method is effective.