Point set registration is a key method in computer vision and pattern recognition. In this paper, the correntropy and bi-directional distance are introduced into registration framework and a new robust registration model for RGB-D data is proposed. Firstly, as registering point sets with smooth structure, such as surface or plane, is easy failed, the color and position information is fused to establish more precise correspondence between two RGB-D data sets. Secondly, to reduce the influence of noises and eliminate outliers, the registration model based on the maximum correntropy criterion is established. Thirdly, the bi-directional distance measurement is introduced into the registration framework to avoid the model being trapped into local extremum. In addition, to solve this new registration problem, a new iterative closest point (ICP) algorithm is proposed, which converges to the local optimal solution by iterations. In the experiments, the proposed algorithm achieves more robustness and precise registration results than other algorithms.