Defect detection is a very important topic in industrial inspection. Due to the rapid expansion of computer computing power, manual-based defect detection approaches is no longer satisfactory, and deep learning-based detection techniques are increasingly being applied to defect detection tasks. contrastive learning is better for defect detection tasks due to the advantages of its label-free training method. So the Efficient Fabric Defect Detector (EFDD), a high-efficiency defect detector based on contrastive learning approach and layered fusion network, is proposed in our paper. First, we propose the Layered Fusion network to encode defect types with different scale sizes and improve the feature extraction ability of the model. Second, a universal structure with an attention module is introduced, this module can be integrated into ResNet with little overhead. Third, we investigate the impact of parallel enhancement strategies on model training in the data imbalance, different combinations can affect the model to different degrees. Finally, we evaluate the model using three classical public datasets and the average improvement of EFDD over the three datasets is AP 2.1%, AR 1.5% compared to other contrastive learning methods. Compared to the supervised methods, our model accuracy is close to or even exceeds that of the supervised methods.