Fabric selection plays an important role in fashion garment design. Designers often use both physical and normalized linguistic criteria for fabric selection. Perception and preference of consumers in their specific sociocultural context, expressed by fashion themes or emotional linguistic criteria, affect greatly new fashion product design. Modeling the relationship between linguistic design criteria and fashion themes of a brand image perceived by consumers becomes thus significant. For setting up this model, we first use fuzzy relations and correlation techniques to select the most relevant linguistic design criteria of fabric hand for each specific fashion theme. The selected criteria can then effectively reduce the complexity of the model and interpret consumer perception of fabrics. Finally, we use a weighted aggregation operator to predict the similarity degree between any new product and fashion themes. Compared with other models, the proposed method is more robust and easier to be interpreted with real data collected for design of senior T-shirt fabrics.