In many small- and medium-sized enterprises (SMEs), defective products are still manually verified in the manufacturing process. Recently, image classification applying deep learning technology has been successful in classifying images of defective and intact products, although there are few cases of utilizing it in practice. SMEs have limited resources; therefore, it is crucial to make careful decisions when applying new methods. We investigated sample size sensitivity to determine the stable performance of deep learning models when applied to the real world. A simple sequential model was constructed, and the dataset was reconstructed into several sizes. For each case, we observed its statistical indicators, such as accuracy, recall, precision, and F1 score, on the same test dataset. Additionally, the loss, accuracy, and AUROC values for the validation dataset were investigated during training. As a result of the conducted research, we were able to confirm that, with 1000 data points or more, the accuracy exceeded 97%. However, more than 5000 cases were required to achieve stability in the model, which had little possibility of overfitting.