Texture represents the surface quality of fabrics, which is one of the key factors for textiles design. Fabric texture can be perceived via sensory perceptions, like vision and touching, causing different psychological feelings and emotions. This paper aimed to model the correlation between fabric textures and the evoked emotions through different sensory perceptions. Firstly, 20 subjects were required to make sensory evaluation on 10 fabric samples, rating fabric textures and the induced emotions via visual and visual-tactile perceptions. Then, the differences in sensory evaluation on fabric textures and the evoked emotions were analyzed. The results showed that only by visual perception, the visual effects dominate the evaluation on fabric textures and emotions. Pile fabrics that look warm can induce positive emotions. By visual-tactile perception, the sensation of contact between fabric and skin accounts due to the intervention of tactile perception. Soft and smooth fabrics, like coral fleece and velvet can stimulate positive emotions. Finally, the emotion-fabric textures model for pleasure and arousal was established using stepwise regression analysis. It’s revealed that only by visual perception, cold/warm had a significant positive effect on pleasure, while no textures were found that have a significant impact on arousal. By visual-tactile perception, both hard/soft and rough/smooth had significant positive impact on pleasure, while thick/thin had a significant negative influence on arousal. The present findings can help direct textiles design, production and sales, so as to meet the specific requirements.