The defect detection is an important activity in quality analysis and control in the fabric industry. The presented work gives a comparative analysis of artificial neural network and deep learning architectures. The MobileNet and deep residual network (ResNet) are deployed to classify the defective and nondefective fabric images. The hand-crafted morphological features are used in fabric image analysis along with feed backward selection feature reduction method to obtain the significant features. The overall classification rates of 95.3%, 98.2%, and 99.65% are obtained for Shallow, ResNet, and MobileNet architectures, respectively. The MobileNet model has given a maximum classification rate than Shallow and ResNet architectures. The work finds applications in apparel industry, quality analysis, cost estimation, online purchase of fabric, Industry 4.0, and so on.