A fast Coding Unit (CU) partition decision strategy based on Human Visual System (HVS) perception quality is proposed in this paper. Considering that it is difficult for existing fast algorithms to further improve compression efficiency, perceptual coding technology has been tried to remove visual redundancy for achieving the purpose of reducing the bit rate on the basis of maintaining subjective visual quality. However, the existing perceptual coding model is still insufficient to reflect the characteristics of the HVS, which has limited improvement in coding efficiency, especially in fast algorithms. In this method, the characteristics of the limited human capacity for spatial-temporal resolution and the visual sensory memory are used to improve coding performance. First, the color complexity is used as a control factor to optimize the Just Noticeable Difference (JND) model to remove visual redundancy in a way that is more in line with the characteristics of visual perception. Second, a classification model of motion patterns based on human visual saliency is designed to provide a basis for CU classification, which effectively improves the coding accuracy. Finally, an offline Decision Tree (DT) classifier is designed based on the above model, and texture features are incorporated into the classifier as another key attribute to further reduce the computational complexity. The results of performance evaluation confirm that the proposed method achieves significantly improved coding performance compared with original Versatile Test Model (VTM). Compared with existing algorithms, our method not only improves coding efficiency, but also improves subjective visual quality.