Presently, the pearl quality inspection mainly relies on manual. In this paper, we collected the microscopy images of pearl samples and built the novel method of pearl defects detection based on deep convolutional neural network. 1. Identification and classification of pearl defect images: a) 2000 images of 250 pearls were taken by stereomicroscopy as data set; b) The data set was augmented with ImageDataGenerator toolbox; c) The impact of overfitting was reduced by combining Dropout method; d) Based on VGG-16 model, feature extraction and fine-tuning methods were adopted to achieve the ideal recognition and classification effects.2. Pearl defect area calculation: a) The image was preprocessed using MATLAB software, including color space conversion, image filtering and threshold segmentation; b) In order to obtain clear, highly differentiated and continuous contour images, the improved Sobel operator was used for edge detection; c) The pearl defect area in a single plane was obtained by inverse edge detection.