Because of the subjectivity of human beings, the evaluation of image complexity in the Human Vision System (HVS) cannot be provided accurately by traditional image complexity evaluation models. In the 2016 Conference on Computer Vision and Pattern Recognition (CVPR 2016), an evaluation method of visual search difficulty based on the visual search time was proposed for the first time. In this paper, the ordinal relation of the image complexity for human perception was discussed, and a quantitative evaluation model based on Convolutional Neural Network (CNN) features and Support Vector Ordinal Regression (SVOR) with explicit inequality constraints on the thresholds was proposed. The results showed that the evaluation models based on SVOR and pyramid CNN features of images can describe the order relation of image complexity among different images more accurately, which achieve the Kendall’s tau correlation of 0.4858, better than SVR overall under the same condition, whose highest Kendall’s tau correlation is 0.4794.