The leather industry is increasingly becoming one amongst the most important manufacturing industries in the world. Increasing demand has posed a great challenge as well as an opportunity for these industries. Quality of a leather product has been always the main factor in the setting of the market selling price. Usually, quality control is done with manual inspection. However, with human related errors such as fatigue, loss of concentration, etc., misclassification of the produced leather quality becomes a very serious issue. To tackle this issue, traditionally, image processing algorithms have been used, but, have not been effective due to low accuracies and high processing time. The introduction of Deep Learning methodologies such as Convolutional Neural Networks (CNNs), however, makes image classification much simpler. It incorporates automated feature learning and extraction, giving accurate results in lesser time. In addition, the usage of deep learning can also be applied for defect detection, which is, locating defects in the image. In this paper, a system for leather image classification and defect detection is proposed. Initially, the captured images are sent to a classification system, which classifies the image as good quality or defect quality. If the output of the classification system is defect quality, then a defect detection system works on the images, and locates the defects in the image. The classification system and the defect detection system are developed using Inception V3 CNN and Mask R-CNN respectively. Experimental results using these CNNs have shown great potential with respect to object classification and detection, which, with further development can give unparalleled performance for applications in these fields.